-----============= acceptance-small: sanity-flr ============----- Mon Jun 1 12:31:36 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg113-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 12:31:55 (1780331515) === oleg113-client.virtnet: executing check_config_client /mnt/lustre oleg113-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg113-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff960286164800.idle_timeout=debug osc.lustre-OST0001-osc-ffff960286164800.idle_timeout=debug disable quota as required oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:32:31 (1780331551) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8225] == sanity-flr test 0a: lfs mirror create with -N option == 12:32:33 (1780331553) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:32:51 (1780331571) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:32:55 (1780331575) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:32:58 (1780331578) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:33:21 (1780331601) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:33:24 (1780331604) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 12:33:28 (1780331608) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff960286164800.stats=clear osc.lustre-OST0001-osc-ffff960286164800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:33:35 (1780331615) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:33:46 (1780331626) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:33:53 (1780331633) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:33:56 (1780331636) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:34:04 (1780331644) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 12:34:14 (1780331654) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:34:21 (1780331661) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:34:29 (1780331669) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:34:38 (1780331678) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:34:47 (1780331687) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:34:53 (1780331693) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.373154 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781659 s, 13.4 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:35:01 (1780331701) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.133067 s, 39.4 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:35:08 (1780331708) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:35:18 (1780331718) Stopping /mnt/lustre-ost1 (opts:) on oleg113-server oleg113-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff960286164800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff960286164800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg113-server: oleg113-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg113-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg113-server oleg113-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff960286164800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff960286164800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg113-server: oleg113-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg113-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (61s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:36:19 (1780331779) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.349733 s, 9.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg113-server oleg113-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff960286164800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff960286164800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg113-server: oleg113-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg113-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:37:02 (1780331822) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg113-server oleg113-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff960286164800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff960286164800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg113-server: oleg113-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg113-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg113-server oleg113-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff960286164800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff960286164800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg113-server: oleg113-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg113-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:38:22 (1780331902) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.59718 s, 20.2 MB/s -rw-r--r-- 1 root root 52428800 Jun 1 12:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.888s user 0m0.000s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg113-server oleg113-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff960286164800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff960286164800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.164s user 0m0.001s sys 0m0.192s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg113-server: oleg113-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg113-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg113-server oleg113-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff960286164800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff960286164800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.227s user 0m0.002s sys 0m0.202s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg113-server: oleg113-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg113-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg113-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff960286164800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff960286164800.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:39:53 (1780331993) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:39:57 (1780331997) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:40:02 (1780332002) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 12:40:06 (1780332006) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 12:40:15 (1780332015) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.28676 s, 43.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:40:27 (1780332027) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416213 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236947 s, 44.3 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:40:35 (1780332035) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:40:44 (1780332044) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00128688 s, 796 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111147 s, 921 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102987 s, 994 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00106239 s, 964 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 12:40:55 (1780332055) create test files with size 15749714 .. e615b3ca03bf2624d946d0039d383e0d - fccdced89f6d6dc9727688b6e3d8f32a - a1b67ba22be3f19807335d5e6b9bc694 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg113-client.virtnet /mnt/lustre (opts:) Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre PASS 37 (31s) == sanity-flr test 38: resync ============================ 12:41:26 (1780332086) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:41:42 (1780332102) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:41:49 (1780332109) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087422 s, 12.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416835 s, 25.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 12:41:59 (1780332119) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0766795 s, 27.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.396738 s, 10.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13369 s, 15.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 12:42:15 (1780332135) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 12:42:18 (1780332138) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:42:22 (1780332142) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592466 s, 17.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 12:42:31 (1780332151) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.369826 s, 8.5 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:42:42 (1780332162) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:42:54 (1780332174) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.474642 s, 22.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg113-client.virtnet /mnt/lustre (opts:) Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:43:09 (1780332189) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff96028799f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.328485 s, 31.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg113-client.virtnet /mnt/lustre (opts:) Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff960292076000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:43:25 (1780332205) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:44:03 (1780332243) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:44:44 (1780332284) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:44:48 (1780332288) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.275544 s, 11.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.198687 s, 15.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x95a28e24 (nosync,stale) Mirror 2: 0x95a28e24 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1a9be140abf3164eae08d20e7ab702ca - mirror 1 checksum: d686751aa599168f2237d563322d8cd5 - mirror 2 checksum: 1a9be140abf3164eae08d20e7ab702ca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 12:44:50 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Jun 1 12:44:50 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1a9be140abf3164eae08d20e7ab702ca - mirror 1 checksum: d686751aa599168f2237d563322d8cd5 - mirror 2 checksum: d686751aa599168f2237d563322d8cd5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 12:44:50 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Jun 1 12:44:50 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:45:00 (1780332300) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428163 s, 24.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:45:08 (1780332308) Failing mds1 on oleg113-server Stopping /mnt/lustre-mds1 (opts:) on oleg113-server 12:45:20 (1780332320) shut down facet: mds1 facet_host: oleg113-server facet_failover_host: oleg113-server Failover mds1 to oleg113-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg113-server: oleg113-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg113-client: oleg113-server: ssh exited with exit code 1 Started lustre-MDT0000 12:45:53 (1780332353) targets are mounted 12:45:53 (1780332353) facet_failover done oleg113-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486319 s, 21.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:46:09 (1780332369) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0272469 s, 752 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.015259 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:46:19 (1780332379) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0741653 s, 276 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0260279 s, 787 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:46:30 (1780332390) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0245136 s, 668 kB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:46:40 (1780332400) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0379185 s, 540 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:46:50 (1780332410) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0098044 s, 418 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0201206 s, 204 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:46:56 (1780332416) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:47:21 (1780332441) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:47:51 (1780332471) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:48:16 (1780332496) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:48:41 (1780332521) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 12:48:49 (1780332529) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57162 mirror split pid 57164 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57162 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 57164 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 12:49:57 (1780332597) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 2600 000001 1780332600.652192 mapwrite 0x02331b thru 0x02e9fd (0xb6e3 bytes) 000002 1780332600.805731 punch from 0x02c668 to 0x02e9fd (0x2396 bytes) truncating to largest ever: 0x1ddcb 000003 1780332600.820098 trunc from 0x02e9fe to 0x01ddca (0x10c33 bytes) fallocating to largest ever: 0x40000 000005 1780332600.857326 fallocate 0x031519 thru 0x03ffff (0xeae7 bytes) 000006 1780332600.877450 write 0x0128f1 thru 0x01d43e (0xab4e bytes) 000007 1780332600.907542 mapread 0x00670a thru 0x0099ad (0x32a4 bytes) 000008 1780332600.931017 punch from 0x00eab5 to 0x01b65d (0xcba9 bytes) 000010 1780332600.984155 mapwrite 0x027902 thru 0x02ac50 (0x334f bytes) 000012 1780332601.030147 trunc from 0x040000 to 0x020766 (0x1f899 bytes) 000013 1780332601.076647 read 0x01beb2 thru 0x01c9c6 (0xb15 bytes) 000014 1780332601.083946 mapwrite 0x02d03b thru 0x035856 (0x881c bytes) 000015 1780332601.140824 write 0x01bb1a thru 0x024283 (0x876a bytes) 000016 1780332601.149032 trunc from 0x009f53 to 0x011286 (0x7334 bytes) 000017 1780332601.156677 write 0x01631f thru 0x022b7d (0xc85f bytes) 000018 1780332601.177943 MIRROR_EXTEND to 2 mirrors 000019 1780332601.972944 trunc from 0x035857 to 0x015b7f (0x1fcd7 bytes) 000020 1780332602.106670 trunc from 0x015b80 to 0x0327fb (0x1cc7c bytes) 000021 1780332602.152769 trunc from 0x0327fc to 0x01e725 (0x140d6 bytes) 000022 1780332602.180600 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e726): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e726): Mirror 2: 0x44217874 (stale) CRC-32 checksum value for chunk [0, 0x1e726): Mirror 1: 0xdad3f3cd Mirror 3: 0xdad3f3cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e726: skipped 000023 1780332603.452582 trunc from 0x01e726 to 0x02c490 (0xdd6b bytes) 000024 1780332603.537810 trunc from 0x02c491 to 0x00a0f3 (0x2239d bytes) 000025 1780332603.564130 trunc from 0x00a0f4 to 0x0352bf (0x2b1cc bytes) 000026 1780332603.584031 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x352c0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x352c0): Mirror 2: 0x99866129 (stale) Mirror 3: 0x6648cc38 (stale) CRC-32 checksum value for chunk [0, 0x352c0): Mirror 1: 0x6648cc38 Mirror 4: 0x6648cc38 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x352c0: skipped 000027 1780332604.667241 trunc from 0x0352c0 to 0x006641 (0x2ec7e bytes) 000028 1780332604.730854 mapread 0x0014fa thru 0x006641 (0x5148 bytes) 000030 1780332604.748786 mapread 0x005db1 thru 0x006641 (0x891 bytes) 000031 1780332604.758314 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6642): Mirror 2: 0xfa5a5581 (stale) Mirror 4: 0xfa5a5581 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6642): Mirror 2: 0xfa5a5581 (stale) Mirror 4: 0xfa5a5581 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6642: skipped 000032 1780332605.108674 mapread 0x003089 thru 0x004e6c (0x1de4 bytes) 000033 1780332605.121585 fallocate 0x002a98 thru 0x00d4f8 (0xaa61 bytes) 000034 1780332605.127514 trunc from 0x00354f to 0x003f2a (0x9dc bytes) 000035 1780332605.132306 read 0x004f49 thru 0x006641 (0x16f9 bytes) 000036 1780332605.137524 mapwrite 0x01e66e thru 0x02c7bd (0xe150 bytes) 000037 1780332605.199978 fallocate 0x012696 thru 0x01781a (0x5185 bytes) 000038 1780332605.206647 trunc from 0x0298c5 to 0x02c7bd (0x2ef9 bytes) 000039 1780332605.211937 mapread 0x01e3c0 thru 0x0272ff (0x8f40 bytes) 000040 1780332605.213909 fallocate 0x00fbab thru 0x013a0a (0x3e60 bytes) 000041 1780332605.217962 fallocate 0x0160f9 thru 0x022560 (0xc468 bytes) 000042 1780332605.221737 write 0x03b0f3 thru 0x03bfcf (0xedd bytes) 000043 1780332605.222678 read 0x0193e8 thru 0x0196b9 (0x2d2 bytes) 000044 1780332605.241411 MIRROR_RESYNC 3 mirrors 000045 1780332606.543057 write 0x03d905 thru 0x03eddf (0x14db bytes) 000046 1780332606.587073 mapread 0x002e8e thru 0x011416 (0xe589 bytes) 000047 1780332606.632166 mapwrite 0x02c421 thru 0x039f41 (0xdb21 bytes) 000048 1780332606.726055 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3ede0): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ede0): Mirror 2: 0x1241687f (stale) Mirror 4: 0x1241687f (stale) CRC-32 checksum value for chunk [0, 0x3ede0): Mirror 1: 0xa7bd61ae Mirror 5: 0xa7bd61ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ede0: skipped 000049 1780332607.897076 MIRROR_RESYNC 4 mirrors 000050 1780332609.267720 trunc from 0x03d779 to 0x03eddf (0x1667 bytes) 000051 1780332609.343907 mapread 0x038342 thru 0x03b02f (0x2cee bytes) 000052 1780332609.391652 fallocate 0x00cf3a thru 0x00e43a (0x1501 bytes) 000053 1780332609.409543 write 0x01e84a thru 0x023ae9 (0x52a0 bytes) 000054 1780332609.437151 trunc from 0x0077cf to 0x00ac2b (0x345d bytes) 000055 1780332609.443115 trunc from 0x03ede0 to 0x038dfd (0x5fe2 bytes) 000056 1780332609.481176 fallocate 0x01b9b0 thru 0x0201d1 (0x4822 bytes) 000057 1780332609.488572 trunc from 0x038dfe to 0x02da16 (0xb3e7 bytes) 000058 1780332609.513005 fallocate 0x023961 thru 0x02e679 (0xad19 bytes) 000059 1780332609.521363 trunc from 0x02da17 to 0x03e727 (0x10d11 bytes) 000060 1780332609.545403 mapread 0x02c62e thru 0x030a7d (0x4450 bytes) 000061 1780332609.562450 mapread 0x00b91b thru 0x017c1b (0xc301 bytes) 000062 1780332609.587549 trunc from 0x03e728 to 0x02c3c0 (0x12367 bytes) 000063 1780332609.605344 trunc from 0x02c3c1 to 0x01d8c4 (0xeafc bytes) 000064 1780332609.631547 trunc from 0x01d8c5 to 0x01f884 (0x1fc0 bytes) 000065 1780332609.650943 fallocate 0x0281bc thru 0x0362ce (0xe113 bytes) 000066 1780332609.658620 mapread 0x0147f2 thru 0x017c6e (0x347d bytes) 000067 1780332609.660114 fallocate 0x005aac thru 0x00f45e (0x99b3 bytes) 000068 1780332609.665985 read 0x0267cd thru 0x0281ae (0x19e2 bytes) 000069 1780332609.673369 trunc from 0x0362cf to 0x01ef86 (0x17348 bytes) 000070 1780332609.692660 read 0x004be0 thru 0x0056bd (0xade bytes) 000071 1780332609.693983 fallocate 0x03060c thru 0x0370ea (0x6adf bytes) 000072 1780332609.703625 MIRROR_RESYNC 4 mirrors 000073 1780332610.659055 punch from 0x027ea0 to 0x0370ea (0xf24b bytes) 000074 1780332610.703162 MIRROR_RESYNC 4 mirrors 000076 1780332611.681671 mapread 0x02c012 thru 0x0370ea (0xb0d9 bytes) 000078 1780332611.715980 read 0x027e2f thru 0x0370ea (0xf2bc bytes) 000080 1780332611.716999 read 0x001d51 thru 0x002f45 (0x11f5 bytes) 000081 1780332611.744209 MIRROR_RESYNC 4 mirrors 000082 1780332612.320237 read 0x00a877 thru 0x014905 (0xa08f bytes) 000083 1780332612.321905 punch from 0x0044b5 to 0x00d305 (0x8e51 bytes) 000084 1780332612.374057 fallocate 0x035780 thru 0x03ffff (0xa880 bytes) 000086 1780332612.404448 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 6: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000087 1780332613.490331 mapread 0x0104bb thru 0x01a3a1 (0x9ee7 bytes) 000088 1780332613.547991 write 0x0203b9 thru 0x02c9c5 (0xc60d bytes) 000089 1780332613.624678 read 0x0177ff thru 0x01ee12 (0x7614 bytes) 000090 1780332613.630817 mapwrite 0x009625 thru 0x01910c (0xfae8 bytes) 000091 1780332613.719499 trunc from 0x040000 to 0x028077 (0x17f88 bytes) 000092 1780332613.765969 trunc from 0x00f560 to 0x01ed6f (0xf810 bytes) 000093 1780332613.781868 MIRROR_RESYNC 5 mirrors 000094 1780332615.451119 fallocate 0x02141a thru 0x0297f6 (0x83dd bytes) 000095 1780332615.501129 fallocate 0x025dc6 thru 0x029bbe (0x3df9 bytes) 000096 1780332615.507009 mapwrite 0x03a0c9 thru 0x03c426 (0x235e bytes) 000097 1780332615.591025 punch from 0x0149be to 0x0166d2 (0x1d15 bytes) 000098 1780332615.600708 mapwrite 0x013e51 thru 0x0184c8 (0x4678 bytes) 000099 1780332615.660877 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c427): Mirror 2: 0x15d1dd1 (stale) Mirror 4: 0x15d1dd1 (stale) Mirror 6: 0x15d1dd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c427): Mirror 2: 0x15d1dd1 (stale) Mirror 4: 0x15d1dd1 (stale) Mirror 6: 0x15d1dd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c427: skipped 000100 1780332616.127852 fallocate 0x015936 thru 0x022a1e (0xd0e9 bytes) 000101 1780332616.134811 trunc from 0x023831 to 0x0325ce (0xed9e bytes) 000102 1780332616.142787 punch from 0x02c6fd to 0x02ce43 (0x747 bytes) 000103 1780332616.149720 mapwrite 0x00cef8 thru 0x00f9fd (0x2b06 bytes) 000104 1780332616.188322 read 0x034d73 thru 0x03b0a4 (0x6332 bytes) 000105 1780332616.202780 fallocate 0x005210 thru 0x005499 (0x28a bytes) 000106 1780332616.208774 punch from 0x021b7f to 0x030080 (0xe502 bytes) 000107 1780332616.216692 trunc from 0x03c427 to 0x0167e3 (0x25c43 bytes) 000108 1780332616.242097 punch from 0x00d412 to 0x010b50 (0x373f bytes) 000109 1780332616.249934 read 0x008e91 thru 0x00eb0b (0x5c7b bytes) 000110 1780332616.266659 trunc from 0x0167e4 to 0x018c06 (0x2423 bytes) 000111 1780332616.284167 punch from 0x012385 to 0x018c06 (0x6882 bytes) 000113 1780332616.291827 mapwrite 0x017dc3 thru 0x01f2fc (0x753a bytes) 000114 1780332616.337732 mapwrite 0x01a685 thru 0x027f24 (0xd8a0 bytes) 000116 1780332616.409056 mapread 0x004e76 thru 0x00efbd (0xa148 bytes) 000117 1780332616.417483 MIRROR_RESYNC 4 mirrors 000118 1780332617.416758 read 0x01039d thru 0x0190fa (0x8d5e bytes) 000119 1780332617.446293 trunc from 0x008104 to 0x0115cf (0x94cc bytes) 000120 1780332617.504371 write 0x007c36 thru 0x00e7cd (0x6b98 bytes) 000121 1780332617.515874 write 0x024f28 thru 0x02f6ec (0xa7c5 bytes) 000122 1780332617.544078 read 0x001ee0 thru 0x00d55b (0xb67c bytes) 000123 1780332617.577995 read 0x018675 thru 0x0192b7 (0xc43 bytes) 000124 1780332617.578684 punch from 0x01d0ea to 0x023a7d (0x6994 bytes) 000125 1780332617.594882 trunc from 0x013ee2 to 0x021551 (0xd670 bytes) 000126 1780332617.611239 trunc from 0x02d00c to 0x02f6ec (0x26e1 bytes) 000127 1780332617.639309 trunc from 0x02b025 to 0x02f6ec (0x46c8 bytes) 000128 1780332617.645536 trunc from 0x00b37f to 0x00e6ef (0x3371 bytes) 000129 1780332617.651853 read 0x011b02 thru 0x01d2c2 (0xb7c1 bytes) 000131 1780332617.674753 trunc from 0x00e538 to 0x010257 (0x1d20 bytes) 000132 1780332617.681046 punch from 0x00a474 to 0x0184d1 (0xe05e bytes) 000133 1780332617.688644 mapread 0x0164da thru 0x01ba8b (0x55b2 bytes) 000134 1780332617.705042 mapwrite 0x034b2a thru 0x03f656 (0xab2d bytes) 000135 1780332617.757402 mapwrite 0x01f6af thru 0x021c8e (0x25e0 bytes) 000136 1780332617.776333 punch from 0x039c1c to 0x039ebc (0x2a1 bytes) 000137 1780332617.783864 trunc from 0x02dedd to 0x033057 (0x517b bytes) 000139 1780332617.801836 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f657): Mirror 4: 0xc0a66f6f (stale) Mirror 6: 0xc0a66f6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f657): Mirror 4: 0xc0a66f6f (stale) Mirror 6: 0xc0a66f6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f657: skipped 000140 1780332618.231868 mapwrite 0x020430 thru 0x022a51 (0x2622 bytes) 000141 1780332618.289785 trunc from 0x03f657 to 0x03d6ae (0x1fa8 bytes) 000142 1780332618.344744 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d6af): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d6af): Mirror 4: 0x61104015 (stale) Mirror 6: 0x61104015 (stale) CRC-32 checksum value for chunk [0, 0x3d6af): Mirror 1: 0xdc764b4a Mirror 7: 0xdc764b4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d6af: skipped 000144 1780332619.881104 write 0x003f45 thru 0x013ca7 (0xfd63 bytes) 000145 1780332619.936352 mapread 0x02c60e thru 0x03b957 (0xf34a bytes) 000146 1780332619.962214 trunc from 0x03d6af to 0x0337ed (0x9ec1 bytes) 000147 1780332619.987698 mapread 0x0273b7 thru 0x02f295 (0x7edf bytes) 000148 1780332619.990166 trunc from 0x0337ee to 0x013fcc (0x1f821 bytes) 000149 1780332620.012992 read 0x0002fb thru 0x00887d (0x8583 bytes) 000150 1780332620.022932 write 0x01a440 thru 0x025b8d (0xb74e bytes) 000151 1780332620.025918 mapread 0x004745 thru 0x00c21c (0x7ad8 bytes) 000152 1780332620.027649 mapwrite 0x029a4c thru 0x02b996 (0x1f4b bytes) 000153 1780332620.081797 write 0x01a083 thru 0x01de66 (0x3de4 bytes) 000154 1780332620.084356 trunc from 0x01da0c to 0x025006 (0x75fb bytes) 000155 1780332620.104706 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b997): Mirror 6: 0xeef14c9c (stale) Mirror 7: 0xe9483aae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b997): Mirror 6: 0xeef14c9c (stale) Mirror 7: 0xe9483aae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b997: skipped 000156 1780332620.681069 mapread 0x019e9b thru 0x01d3b0 (0x3516 bytes) 000157 1780332620.707471 trunc from 0x004237 to 0x004a6e (0x838 bytes) 000158 1780332620.712581 trunc from 0x0024c5 to 0x008c0d (0x6749 bytes) 000159 1780332620.717932 fallocate 0x025641 thru 0x02abec (0x55ac bytes) 000161 1780332620.726209 mapwrite 0x024df7 thru 0x028879 (0x3a83 bytes) 000162 1780332620.758362 mapread 0x0006e6 thru 0x00c4b6 (0xbdd1 bytes) 000163 1780332620.772821 write 0x03ea47 thru 0x03ffff (0x15b9 bytes) 000164 1780332620.784199 mapread 0x039bae thru 0x03ffff (0x6452 bytes) 000165 1780332620.808186 mapwrite 0x03ad2d thru 0x03ffff (0x52d3 bytes) 000166 1780332620.832384 trunc from 0x011ec1 to 0x01e3be (0xc4fe bytes) 000167 1780332620.845078 MIRROR_RESYNC 3 mirrors 000168 1780332622.058407 mapwrite 0x004752 thru 0x012822 (0xe0d1 bytes) 000169 1780332622.186793 write 0x017934 thru 0x01baa0 (0x416d bytes) 000171 1780332622.199282 trunc from 0x040000 to 0x00ff3d (0x300c2 bytes) 000172 1780332622.238863 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xff3e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xff3e): Mirror 6: 0x6f5c5cd2 (stale) Mirror 7: 0x6f5c5cd2 (stale) CRC-32 checksum value for chunk [0, 0xff3e): Mirror 1: 0x8fb2726b Mirror 8: 0x8fb2726b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xff3e: skipped 000173 1780332623.407983 write 0x018661 thru 0x025a80 (0xd420 bytes) 000175 1780332623.479099 mapread 0x012196 thru 0x016783 (0x45ee bytes) 000176 1780332623.505158 mapwrite 0x011a83 thru 0x0186ba (0x6c38 bytes) 000177 1780332623.559422 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25a81): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25a81): Mirror 6: 0x42e94262 (stale) Mirror 7: 0x42e94262 (stale) Mirror 8: 0xbf767ba6 (stale) CRC-32 checksum value for chunk [0, 0x25a81): Mirror 1: 0xfde6580b Mirror 9: 0xfde6580b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25a81: skipped 000178 1780332625.149214 trunc from 0x01ddca to 0x025a80 (0x7cb7 bytes) 000179 1780332625.274039 mapread 0x007823 thru 0x007a94 (0x272 bytes) 000181 1780332625.294359 fallocate 0x015b6b thru 0x019d90 (0x4226 bytes) 000182 1780332625.314137 mapread 0x01ed26 thru 0x025a80 (0x6d5b bytes) 000183 1780332625.353406 mapread 0x01b90d thru 0x025a80 (0xa174 bytes) 000184 1780332625.354969 read 0x017c26 thru 0x01f7f6 (0x7bd1 bytes) 000185 1780332625.368017 trunc from 0x0082f3 to 0x010579 (0x8287 bytes) 000186 1780332625.377746 trunc from 0x0230a9 to 0x025a80 (0x29d8 bytes) 000187 1780332625.385894 write 0x033c32 thru 0x03ffff (0xc3ce bytes) 000188 1780332625.398671 mapwrite 0x00d2ec thru 0x01b93f (0xe654 bytes) 000189 1780332625.501557 read 0x0180e9 thru 0x01960c (0x1524 bytes) 000190 1780332625.502216 trunc from 0x03f52a to 0x03ffff (0xad6 bytes) 000191 1780332625.509236 trunc from 0x040000 to 0x0350a2 (0xaf5d bytes) 000192 1780332625.540993 fallocate 0x022ac5 thru 0x02ca30 (0x9f6c bytes) 000193 1780332625.551238 read 0x01e624 thru 0x02a3ca (0xbda7 bytes) 000194 1780332625.565870 trunc from 0x0350a3 to 0x02b522 (0x9b80 bytes) 000195 1780332625.595960 mapread 0x00e743 thru 0x016779 (0x8037 bytes) 000196 1780332625.597189 punch from 0x019195 to 0x01cc09 (0x3a75 bytes) 000197 1780332625.613056 punch from 0x01b4e9 to 0x023c87 (0x879f bytes) 000198 1780332625.623654 punch from 0x02576e to 0x02b522 (0x5db5 bytes) 000199 1780332625.639738 punch from 0x0197ab to 0x021f46 (0x879c bytes) 000201 1780332625.650088 mapwrite 0x00575e thru 0x009707 (0x3faa bytes) 000202 1780332625.692006 read 0x0086cb thru 0x013aa2 (0xb3d8 bytes) 000203 1780332625.693029 mapwrite 0x02c6b3 thru 0x030d43 (0x4691 bytes) 000204 1780332625.810713 read 0x02af1e thru 0x030d43 (0x5e26 bytes) 000206 1780332625.812011 mapread 0x00f0a4 thru 0x018d40 (0x9c9d bytes) 000207 1780332625.837065 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30d44): Mirror 6: 0x519c66cf (stale) Mirror 7: 0x519c66cf (stale) Mirror 8: 0xc14c5043 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d44): Mirror 6: 0x519c66cf (stale) Mirror 7: 0x519c66cf (stale) Mirror 8: 0xc14c5043 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d44: skipped 000208 1780332626.355546 MIRROR_RESYNC 4 mirrors 000209 1780332627.918544 trunc from 0x030d44 to 0x03761e (0x68db bytes) 000210 1780332628.011517 punch from 0x01439b to 0x01fb5a (0xb7c0 bytes) 000211 1780332628.018287 fallocate 0x0232d8 thru 0x032491 (0xf1ba bytes) 000212 1780332628.024012 mapwrite 0x01cf09 thru 0x026852 (0x994a bytes) 000213 1780332628.102200 read 0x01517b thru 0x023fe1 (0xee67 bytes) 000214 1780332628.117580 trunc from 0x00805a to 0x008f77 (0xf1e bytes) 000215 1780332628.131484 mapread 0x02d8c6 thru 0x034b0b (0x7246 bytes) 000216 1780332628.157035 fallocate 0x001243 thru 0x008618 (0x73d6 bytes) 000217 1780332628.165038 trunc from 0x01b992 to 0x01e718 (0x2d87 bytes) 000218 1780332628.176340 fallocate 0x0183f5 thru 0x01dace (0x56da bytes) 000219 1780332628.193082 trunc from 0x03761f to 0x03acbd (0x369f bytes) 000220 1780332628.254065 read 0x031666 thru 0x03a8c5 (0x9260 bytes) 000221 1780332628.277707 trunc from 0x012ff2 to 0x01e775 (0xb784 bytes) 000222 1780332628.289091 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3acbe): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3acbe): Mirror 6: 0xc9a40ca7 (stale) Mirror 7: 0xc9a40ca7 (stale) Mirror 8: 0xc9a40ca7 (stale) CRC-32 checksum value for chunk [0, 0x3acbe): Mirror 1: 0x1503c271 Mirror 9: 0x1503c271 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3acbe: skipped 000223 1780332629.445201 MIRROR_RESYNC 5 mirrors 000224 1780332631.016773 fallocate 0x03463b thru 0x036ad9 (0x249f bytes) 000225 1780332631.064186 fallocate 0x03db80 thru 0x03ffff (0x2480 bytes) 000226 1780332631.072112 punch from 0x035eab to 0x038f96 (0x30ec bytes) 000227 1780332631.078439 punch from 0x00e80a to 0x016f4d (0x8744 bytes) 000228 1780332631.084141 trunc from 0x040000 to 0x024af1 (0x1b50e bytes) 000229 1780332631.137596 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24af2): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24af2): Mirror 6: 0x55709f37 (stale) Mirror 7: 0x55709f37 (stale) Mirror 8: 0x55709f37 (stale) Mirror 9: 0x55709f37 (stale) CRC-32 checksum value for chunk [0, 0x24af2): Mirror 1: 0xdbcab45f Mirror 10: 0xdbcab45f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24af2: skipped 000230 1780332632.211760 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x24af2): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24af2): Mirror 6: 0x55709f37 (stale) Mirror 7: 0x55709f37 (stale) Mirror 8: 0x55709f37 (stale) Mirror 9: 0x55709f37 (stale) CRC-32 checksum value for chunk [0, 0x24af2): Mirror 1: 0xdbcab45f Mirror 10: 0xdbcab45f Mirror 11: 0xdbcab45f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24af2: skipped 000231 1780332633.166617 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24af2): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24af2): Mirror 6: 0x55709f37 (stale) Mirror 7: 0x55709f37 (stale) Mirror 8: 0x55709f37 (stale) Mirror 9: 0x55709f37 (stale) CRC-32 checksum value for chunk [0, 0x24af2): Mirror 1: 0xdbcab45f Mirror 10: 0xdbcab45f Mirror 11: 0xdbcab45f Mirror 12: 0xdbcab45f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24af2: skipped 000232 1780332634.142415 mapread 0x0127b7 thru 0x01a2f1 (0x7b3b bytes) 000233 1780332634.164836 write 0x0396a6 thru 0x03ffff (0x695a bytes) 000234 1780332634.196232 mapwrite 0x033441 thru 0x03e095 (0xac55 bytes) 000235 1780332634.224704 write 0x02ff46 thru 0x032668 (0x2723 bytes) 000236 1780332634.226883 read 0x00c12b thru 0x0128e5 (0x67bb bytes) 000237 1780332634.233534 mapwrite 0x03fc78 thru 0x03ffff (0x388 bytes) 000238 1780332634.262507 read 0x00ba71 thru 0x00d1e8 (0x1778 bytes) 000239 1780332634.278430 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x10c77edf (stale) Mirror 7: 0x10c77edf (stale) Mirror 8: 0x10c77edf (stale) Mirror 9: 0x10c77edf (stale) Mirror 11: 0x7b11441e (stale) Mirror 12: 0x7b11441e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x10c77edf (stale) Mirror 7: 0x10c77edf (stale) Mirror 8: 0x10c77edf (stale) Mirror 9: 0x10c77edf (stale) Mirror 11: 0x7b11441e (stale) Mirror 12: 0x7b11441e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000240 1780332634.997043 read 0x01465f thru 0x01936b (0x4d0d bytes) 000241 1780332635.016723 mapread 0x008143 thru 0x015477 (0xd335 bytes) 000242 1780332635.051504 read 0x032b95 thru 0x0337b6 (0xc22 bytes) 000243 1780332635.063352 mapwrite 0x021fd9 thru 0x0246c2 (0x26ea bytes) 000244 1780332635.079629 trunc from 0x0244ea to 0x02ce90 (0x89a7 bytes) 000245 1780332635.085958 trunc from 0x040000 to 0x02d14c (0x12eb3 bytes) 000246 1780332635.120130 fallocate 0x039344 thru 0x03e531 (0x51ee bytes) 000247 1780332635.134511 write 0x01218a thru 0x0193ca (0x7241 bytes) 000249 1780332635.138511 fallocate 0x036327 thru 0x03cdb1 (0x6a8b bytes) 000250 1780332635.145660 read 0x037cd2 thru 0x03cdb1 (0x50e0 bytes) 000252 1780332635.164797 mapread 0x033b72 thru 0x03cdb1 (0x9240 bytes) 000253 1780332635.167006 fallocate 0x01d434 thru 0x0218f6 (0x44c3 bytes) 000254 1780332635.174195 mapwrite 0x02f9e9 thru 0x03a3b8 (0xa9d0 bytes) 000255 1780332635.225045 read 0x008985 thru 0x00baa8 (0x3124 bytes) 000256 1780332635.226528 trunc from 0x03cdb2 to 0x03802f (0x4d82 bytes) 000257 1780332635.249307 fallocate 0x028520 thru 0x0321bc (0x9c9d bytes) 000258 1780332635.256146 trunc from 0x038030 to 0x030768 (0x78c7 bytes) 000260 1780332635.277558 read 0x02c899 thru 0x030768 (0x3ed0 bytes) 000261 1780332635.278343 punch from 0x005a09 to 0x015733 (0xfd2b bytes) 000262 1780332635.285995 trunc from 0x030769 to 0x00ff79 (0x207ef bytes) 000263 1780332635.304557 fallocate 0x01adfc thru 0x0285db (0xd7e0 bytes) 000264 1780332635.310959 write 0x023f2a thru 0x030a32 (0xcb09 bytes) 000265 1780332635.325487 read 0x02bc2d thru 0x030a32 (0x4e06 bytes) 000266 1780332635.326430 mapwrite 0x01f07d thru 0x029a44 (0xa9c8 bytes) 000267 1780332635.347408 punch from 0x01aaad to 0x0292b8 (0xe80c bytes) 000268 1780332635.353634 trunc from 0x010a09 to 0x019d32 (0x932a bytes) 000269 1780332635.359703 write 0x035d5b thru 0x036610 (0x8b6 bytes) 000270 1780332635.362128 write 0x030b69 thru 0x03d603 (0xca9b bytes) 000272 1780332635.364912 mapwrite 0x028fbe thru 0x034e99 (0xbedc bytes) 000273 1780332635.399710 write 0x0377bc thru 0x0390e7 (0x192c bytes) 000274 1780332635.401083 write 0x013ffa thru 0x01862e (0x4635 bytes) 000275 1780332635.403716 trunc from 0x01938c to 0x01b394 (0x2009 bytes) 000276 1780332635.409805 trunc from 0x0211b1 to 0x025aa2 (0x48f2 bytes) 000277 1780332635.422936 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d604): Mirror 7: 0x17cadca8 (stale) Mirror 8: 0x17cadca8 (stale) Mirror 9: 0x17cadca8 (stale) Mirror 11: 0x1cd10d4 (stale) Mirror 12: 0x1cd10d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d604): Mirror 7: 0x17cadca8 (stale) Mirror 8: 0x17cadca8 (stale) Mirror 9: 0x17cadca8 (stale) Mirror 11: 0x1cd10d4 (stale) Mirror 12: 0x1cd10d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d604: skipped 000278 1780332636.046234 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d604): Mirror 7: 0x17cadca8 (stale) Mirror 8: 0x17cadca8 (stale) Mirror 9: 0x17cadca8 (stale) Mirror 12: 0x1cd10d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d604): Mirror 7: 0x17cadca8 (stale) Mirror 8: 0x17cadca8 (stale) Mirror 9: 0x17cadca8 (stale) Mirror 12: 0x1cd10d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d604: skipped 000279 1780332636.533875 trunc from 0x03d604 to 0x014d5d (0x288a6 bytes) 000280 1780332636.556945 write 0x0195bf thru 0x01d65e (0x40a0 bytes) 000281 1780332636.558543 trunc from 0x01c3cb to 0x01d65e (0x1294 bytes) 000282 1780332636.575707 fallocate 0x028e24 thru 0x02a1f5 (0x13d2 bytes) 000283 1780332636.580484 trunc from 0x00e4f6 to 0x014c16 (0x6721 bytes) 000284 1780332636.586870 read 0x00e1fa thru 0x019fa5 (0xbdac bytes) 000285 1780332636.598848 trunc from 0x02a1f6 to 0x0390c2 (0xeecd bytes) 000286 1780332636.617818 read 0x026013 thru 0x033a2d (0xda1b bytes) 000287 1780332636.632495 mapread 0x02b852 thru 0x030bc6 (0x5375 bytes) 000288 1780332636.633770 read 0x00ecfe thru 0x0196a8 (0xa9ab bytes) 000289 1780332636.634506 write 0x01e700 thru 0x01f85a (0x115b bytes) 000290 1780332636.646810 mapread 0x02463f thru 0x02495b (0x31d bytes) 000291 1780332636.651941 trunc from 0x0390c3 to 0x028431 (0x10c91 bytes) 000292 1780332636.681802 mapwrite 0x0028bd thru 0x009e66 (0x75aa bytes) 000293 1780332636.706116 punch from 0x00c9cf to 0x01ac1b (0xe24d bytes) 000294 1780332636.711932 write 0x03b680 thru 0x03e63c (0x2fbd bytes) 000295 1780332636.713913 mapwrite 0x0123be thru 0x01b0d1 (0x8d14 bytes) 000296 1780332636.769960 trunc from 0x03e63d to 0x03c50b (0x2131 bytes) 000297 1780332636.800576 write 0x001a42 thru 0x00716b (0x572a bytes) 000298 1780332636.802017 mapread 0x01d565 thru 0x029f97 (0xca33 bytes) 000299 1780332636.805707 trunc from 0x01d3f8 to 0x01fcc2 (0x28cb bytes) 000300 1780332636.819111 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c50c): Mirror 8: 0xdd7725b5 (stale) Mirror 9: 0xdd7725b5 (stale) Mirror 12: 0x2b014559 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c50c): Mirror 8: 0xdd7725b5 (stale) Mirror 9: 0xdd7725b5 (stale) Mirror 12: 0x2b014559 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c50c: skipped 000303 1780332637.361048 trunc from 0x03c50c to 0x00fe1b (0x2c6f0 bytes) 000304 1780332637.387963 MIRROR_RESYNC 4 mirrors 000305 1780332638.819114 mapwrite 0x01616a thru 0x02398a (0xd821 bytes) 000306 1780332638.961863 trunc from 0x02398b to 0x019447 (0xa543 bytes) 000307 1780332639.012388 fallocate 0x03528e thru 0x03787b (0x25ee bytes) 000308 1780332639.030878 trunc from 0x03787c to 0x0363ef (0x148c bytes) 000309 1780332639.059678 fallocate 0x001069 thru 0x003a01 (0x2999 bytes) 000310 1780332639.074200 trunc from 0x01ebc1 to 0x021758 (0x2b98 bytes) 000311 1780332639.084695 punch from 0x008583 to 0x00d533 (0x4fb1 bytes) 000312 1780332639.102313 trunc from 0x0363f0 to 0x03bd08 (0x5919 bytes) 000313 1780332639.130153 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bd09): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd09): Mirror 8: 0x4444efcc (stale) Mirror 9: 0x4444efcc (stale) Mirror 12: 0x4444efcc (stale) CRC-32 checksum value for chunk [0, 0x3bd09): Mirror 1: 0x7b7c53dc Mirror 13: 0x7b7c53dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd09: skipped 000314 1780332640.862407 fallocate 0x03fd0b thru 0x03ffff (0x2f5 bytes) 000315 1780332640.924639 mapwrite 0x01e355 thru 0x02d6c6 (0xf372 bytes) 000316 1780332641.067075 punch from 0x0136b1 to 0x01fe9d (0xc7ed bytes) 000318 1780332641.078941 fallocate 0x01dd31 thru 0x02d21e (0xf4ee bytes) 000319 1780332641.088352 trunc from 0x0261ef to 0x032ac2 (0xc8d4 bytes) 000320 1780332641.108468 MIRROR_RESYNC 5 mirrors 000321 1780332642.707198 read 0x027821 thru 0x030e33 (0x9613 bytes) 000322 1780332642.720184 trunc from 0x040000 to 0x015739 (0x2a8c6 bytes) 000323 1780332642.768161 fallocate 0x0062dc thru 0x00fa9e (0x97c3 bytes) 000324 1780332642.775052 punch from 0x001892 to 0x0029e9 (0x1158 bytes) 000325 1780332642.783236 write 0x030cd2 thru 0x03f830 (0xeb5f bytes) 000326 1780332642.787747 write 0x033809 thru 0x03ffff (0xc7f7 bytes) 000327 1780332642.789308 mapread 0x02f73d thru 0x03c084 (0xc948 bytes) 000328 1780332642.797899 mapread 0x02d703 thru 0x03117e (0x3a7c bytes) 000329 1780332642.812933 fallocate 0x033a37 thru 0x03ffff (0xc5c9 bytes) 000330 1780332642.821222 fallocate 0x00035e thru 0x00769a (0x733d bytes) 000331 1780332642.829505 mapwrite 0x003939 thru 0x006aba (0x3182 bytes) 000332 1780332642.886237 punch from 0x0017aa to 0x00b213 (0x9a6a bytes) 000333 1780332642.895702 fallocate 0x03fd39 thru 0x03ffff (0x2c7 bytes) 000335 1780332642.904945 trunc from 0x040000 to 0x0280f3 (0x17f0c bytes) 000336 1780332642.941943 fallocate 0x025c1e thru 0x026807 (0xbea bytes) 000337 1780332642.957528 MIRROR_RESYNC 5 mirrors 000338 1780332644.146139 trunc from 0x0280f4 to 0x03dbae (0x15abb bytes) 000339 1780332644.185738 read 0x023e41 thru 0x02a8c9 (0x6a89 bytes) 000341 1780332644.195851 punch from 0x010baf to 0x01417b (0x35cd bytes) 000342 1780332644.200537 punch from 0x02bc71 to 0x033e7d (0x820d bytes) 000344 1780332644.208640 trunc from 0x017e32 to 0x020ad8 (0x8ca7 bytes) 000345 1780332644.215275 write 0x03f7e7 thru 0x03ffff (0x819 bytes) 000346 1780332644.221385 trunc from 0x01da0f to 0x023750 (0x5d42 bytes) 000347 1780332644.233003 write 0x002eb8 thru 0x00915f (0x62a8 bytes) 000348 1780332644.271855 punch from 0x029d47 to 0x032665 (0x891f bytes) 000350 1780332644.288839 trunc from 0x0117c7 to 0x012abd (0x12f7 bytes) 000351 1780332644.306977 mapwrite 0x00091d thru 0x000aca (0x1ae bytes) 000352 1780332644.399088 mapwrite 0x03e739 thru 0x03ffff (0x18c7 bytes) 000354 1780332644.467645 mapwrite 0x028ae6 thru 0x0295ff (0xb1a bytes) 000356 1780332644.533024 trunc from 0x00c025 to 0x018a04 (0xc9e0 bytes) 000357 1780332644.541730 write 0x00fb3b thru 0x01d6d4 (0xdb9a bytes) 000358 1780332644.575754 trunc from 0x03c818 to 0x03ffff (0x37e8 bytes) 000359 1780332644.582993 trunc from 0x0348c1 to 0x03f44d (0xab8d bytes) 000360 1780332644.589194 trunc from 0x040000 to 0x03b3cb (0x4c34 bytes) 000361 1780332644.613191 trunc from 0x00df4f to 0x01a994 (0xca46 bytes) 000362 1780332644.627555 mapwrite 0x00e394 thru 0x018e94 (0xab01 bytes) 000363 1780332644.660127 punch from 0x01a710 to 0x027f4d (0xd83e bytes) 000364 1780332644.664494 punch from 0x01fb50 to 0x026165 (0x6616 bytes) 000365 1780332644.673493 read 0x036472 thru 0x038de3 (0x2972 bytes) 000366 1780332644.699994 trunc from 0x0266bc to 0x033a4a (0xd38f bytes) 000367 1780332644.707964 write 0x009b73 thru 0x01953c (0xf9ca bytes) 000368 1780332644.716321 read 0x0297b2 thru 0x030b1b (0x736a bytes) 000370 1780332644.732998 trunc from 0x03b3cc to 0x000148 (0x3b283 bytes) 000371 1780332644.750186 trunc from 0x000149 to 0x022b4e (0x22a06 bytes) 000372 1780332644.781827 MIRROR_RESYNC 5 mirrors 000373 1780332645.797124 read 0x019529 thru 0x01fb88 (0x6660 bytes) 000375 1780332645.813350 fallocate 0x03302c thru 0x034746 (0x171b bytes) 000376 1780332645.870230 trunc from 0x022b4f to 0x0039ec (0x1f162 bytes) 000377 1780332645.924033 mapwrite 0x01421c thru 0x014263 (0x48 bytes) 000378 1780332646.007592 trunc from 0x00f44f to 0x011abe (0x2670 bytes) 000379 1780332646.013461 mapread 0x00cbd8 thru 0x014263 (0x768c bytes) 000382 1780332646.054972 punch from 0x00c2bf to 0x014263 (0x7fa5 bytes) 000383 1780332646.067818 fallocate 0x03a56c thru 0x03ffff (0x5a94 bytes) 000384 1780332646.083428 write 0x0159dc thru 0x01c9b9 (0x6fde bytes) 000385 1780332646.126696 mapread 0x034a46 thru 0x0356e3 (0xc9e bytes) 000386 1780332646.151518 read 0x020894 thru 0x020c67 (0x3d4 bytes) 000387 1780332646.174993 read 0x00f8d7 thru 0x018868 (0x8f92 bytes) 000388 1780332646.177904 punch from 0x013e9c to 0x01586f (0x19d4 bytes) 000389 1780332646.216703 trunc from 0x021d55 to 0x03183d (0xfae9 bytes) 000390 1780332646.229282 trunc from 0x040000 to 0x022620 (0x1d9df bytes) 000392 1780332646.336508 MIRROR_RESYNC 5 mirrors 000393 1780332648.187901 read 0x005bbb thru 0x0132b0 (0xd6f6 bytes) 000394 1780332648.213009 mapread 0x00789c thru 0x015a65 (0xe1ca bytes) 000395 1780332648.235084 mapwrite 0x009a44 thru 0x01488a (0xae47 bytes) 000396 1780332648.395977 mapwrite 0x001ce4 thru 0x0060d7 (0x43f4 bytes) 000397 1780332648.434765 write 0x03852e thru 0x03ffff (0x7ad2 bytes) 000398 1780332648.440942 fallocate 0x037905 thru 0x03ffff (0x86fb bytes) 000399 1780332648.452622 write 0x02bdf5 thru 0x03a06d (0xe279 bytes) 000400 1780332648.485230 trunc from 0x014b72 to 0x01cffd (0x848c bytes) 000401 1780332648.493197 read 0x0394e3 thru 0x03ffff (0x6b1d bytes) 000402 1780332648.494806 trunc from 0x002759 to 0x008474 (0x5d1c bytes) 000404 1780332648.500779 mapwrite 0x005a2f thru 0x00a880 (0x4e52 bytes) 000405 1780332648.570637 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xb4e94715 (stale) Mirror 9: 0xb4e94715 (stale) Mirror 12: 0xb4e94715 (stale) Mirror 13: 0xb4e94715 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x653840ca Mirror 14: 0x653840ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000407 1780332650.360241 mapread 0x0125c1 thru 0x01ff6b (0xd9ab bytes) 000408 1780332650.453952 write 0x0082dd thru 0x00c1ec (0x3f10 bytes) 000409 1780332650.543371 mapread 0x03991d thru 0x03ffff (0x66e3 bytes) 000411 1780332650.566428 read 0x0211fa thru 0x0262c7 (0x50ce bytes) 000412 1780332650.584671 punch from 0x007952 to 0x017285 (0xf934 bytes) 000413 1780332650.616769 read 0x00e1e8 thru 0x01d2f6 (0xf10f bytes) 000414 1780332650.632875 mapread 0x03b613 thru 0x03ffff (0x49ed bytes) 000415 1780332650.634570 fallocate 0x03a227 thru 0x03ffff (0x5dd9 bytes) 000416 1780332650.642455 read 0x00c1c7 thru 0x00dd00 (0x1b3a bytes) 000417 1780332650.646402 trunc from 0x040000 to 0x022fbf (0x1d040 bytes) 000419 1780332650.708691 mapread 0x018667 thru 0x01ff00 (0x789a bytes) 000420 1780332650.722929 fallocate 0x0203c3 thru 0x0259c5 (0x5603 bytes) 000421 1780332650.741546 read 0x00a803 thru 0x00e677 (0x3e75 bytes) 000422 1780332650.753960 mapwrite 0x03ab93 thru 0x03ffff (0x546d bytes) 000423 1780332650.885228 read 0x03df3a thru 0x03ffff (0x20c6 bytes) 000424 1780332650.886036 fallocate 0x02e896 thru 0x033e8a (0x55f5 bytes) 000425 1780332650.898456 trunc from 0x040000 to 0x03420d (0xbdf2 bytes) 000426 1780332650.936162 mapread 0x022ae3 thru 0x031672 (0xeb90 bytes) 000427 1780332650.937859 punch from 0x0068be to 0x0069e0 (0x123 bytes) 000428 1780332650.960640 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3420e): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3420e): Mirror 8: 0x9bb2c1c (stale) Mirror 9: 0x9bb2c1c (stale) Mirror 12: 0x9bb2c1c (stale) Mirror 13: 0x9bb2c1c (stale) Mirror 14: 0xbea414d6 (stale) CRC-32 checksum value for chunk [0, 0x3420e): Mirror 1: 0x5da6ffa4 Mirror 15: 0x5da6ffa4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3420e: skipped 000429 1780332652.421019 trunc from 0x03420e to 0x02f824 (0x49e9 bytes) 000430 1780332652.507214 write 0x00bc78 thru 0x015709 (0x9a92 bytes) 000431 1780332652.527588 mapread 0x002a7e thru 0x0076c2 (0x4c45 bytes) 000432 1780332652.546074 mapwrite 0x00945e thru 0x00cae6 (0x3689 bytes) 000433 1780332652.603522 trunc from 0x02f825 to 0x007b43 (0x27ce1 bytes) 000434 1780332652.678655 fallocate 0x00238b thru 0x0059c0 (0x3636 bytes) 000435 1780332652.701365 write 0x007ec3 thru 0x013087 (0xb1c5 bytes) 000437 1780332652.706977 fallocate 0x01f4fd thru 0x0240ca (0x4bce bytes) 000439 1780332652.729546 fallocate 0x02f53b thru 0x0333c3 (0x3e89 bytes) 000441 1780332652.736467 punch from 0x010e6a to 0x013087 (0x221e bytes) 000442 1780332652.770485 mapread 0x01064b thru 0x013087 (0x2a3d bytes) 000443 1780332652.786519 trunc from 0x013088 to 0x0250a5 (0x1201e bytes) 000444 1780332652.809242 read 0x00c8a7 thru 0x0164f4 (0x9c4e bytes) 000445 1780332652.819109 punch from 0x00d3a6 to 0x01c21f (0xee7a bytes) 000446 1780332652.846867 fallocate 0x027254 thru 0x0347dd (0xd58a bytes) 000447 1780332652.868884 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x347de): Mirror 8: 0xc7e030af (stale) Mirror 9: 0xc7e030af (stale) Mirror 12: 0xc7e030af (stale) Mirror 14: 0xa839fc1c (stale) Mirror 15: 0xe48e9e0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x347de): Mirror 8: 0xc7e030af (stale) Mirror 9: 0xc7e030af (stale) Mirror 12: 0xc7e030af (stale) Mirror 14: 0xa839fc1c (stale) Mirror 15: 0xe48e9e0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x347de: skipped 000448 1780332653.400443 read 0x0044f1 thru 0x009381 (0x4e91 bytes) 000449 1780332653.411350 mapwrite 0x02d2a0 thru 0x0356db (0x843c bytes) 000450 1780332653.476122 read 0x031c27 thru 0x0356db (0x3ab5 bytes) 000451 1780332653.478147 fallocate 0x004c73 thru 0x014211 (0xf59f bytes) 000452 1780332653.485430 punch from 0x01f8d0 to 0x02bfcc (0xc6fd bytes) 000453 1780332653.491766 trunc from 0x0356dc to 0x039e92 (0x47b7 bytes) 000454 1780332653.514585 write 0x0235b2 thru 0x03015c (0xcbab bytes) 000455 1780332653.528514 read 0x021c74 thru 0x02279f (0xb2c bytes) 000456 1780332653.529552 trunc from 0x039e93 to 0x02a729 (0xf769 bytes) 000457 1780332653.558563 fallocate 0x03c904 thru 0x03ffff (0x36fc bytes) 000458 1780332653.567461 trunc from 0x019bed to 0x01efd3 (0x53e7 bytes) 000459 1780332653.578440 trunc from 0x009ecc to 0x00f173 (0x52a8 bytes) 000460 1780332653.586683 trunc from 0x0054c4 to 0x00ac7d (0x57ba bytes) 000461 1780332653.597946 punch from 0x00947a to 0x00bc76 (0x27fd bytes) 000463 1780332653.609015 trunc from 0x040000 to 0x02c3a4 (0x13c5b bytes) 000464 1780332653.671664 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2c3a5): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c3a5): Mirror 8: 0xb7d70e78 (stale) Mirror 9: 0xb7d70e78 (stale) Mirror 12: 0xb7d70e78 (stale) Mirror 14: 0x6a116ce5 (stale) Mirror 15: 0x48d3aa14 (stale) CRC-32 checksum value for chunk [0, 0x2c3a5): Mirror 1: 0xd11a8744 Mirror 16: 0xd11a8744 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c3a5: skipped 000465 1780332655.155835 trunc from 0x02c3a5 to 0x0315ad (0x5209 bytes) 000466 1780332655.275800 trunc from 0x01c432 to 0x022ea4 (0x6a73 bytes) 000467 1780332655.282006 read 0x014b10 thru 0x01ba2b (0x6f1c bytes) 000468 1780332655.296419 mapread 0x0309fd thru 0x0315ad (0xbb1 bytes) 000469 1780332655.320225 MIRROR_RESYNC 7 mirrors 000470 1780332657.377034 write 0x027671 thru 0x0287b6 (0x1146 bytes) 000471 1780332657.424731 fallocate 0x03e6c7 thru 0x03ffff (0x1939 bytes) 000472 1780332657.454130 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xebeb373 (stale) Mirror 9: 0xebeb373 (stale) Mirror 12: 0xebeb373 (stale) Mirror 14: 0xebeb373 (stale) Mirror 16: 0xebeb373 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xebeb373 (stale) Mirror 9: 0xebeb373 (stale) Mirror 12: 0xebeb373 (stale) Mirror 14: 0xebeb373 (stale) Mirror 16: 0xebeb373 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000473 1780332657.929933 punch from 0x0026fc to 0x007fd3 (0x58d8 bytes) 000474 1780332657.935758 mapread 0x038a1e thru 0x03ffff (0x75e2 bytes) 000475 1780332657.960108 punch from 0x000710 to 0x00408b (0x397c bytes) 000477 1780332657.966127 fallocate 0x016ac4 thru 0x01e1f9 (0x7736 bytes) 000478 1780332657.972667 trunc from 0x040000 to 0x0368df (0x9720 bytes) 000479 1780332658.003328 write 0x0233a6 thru 0x02764c (0x42a7 bytes) 000480 1780332658.014316 write 0x031fa7 thru 0x03d594 (0xb5ee bytes) 000481 1780332658.028840 trunc from 0x00d20b to 0x019494 (0xc28a bytes) 000482 1780332658.040687 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d595): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d595): Mirror 8: 0x7ee9cb21 (stale) Mirror 9: 0x7ee9cb21 (stale) Mirror 12: 0x7ee9cb21 (stale) Mirror 14: 0x7ee9cb21 (stale) Mirror 16: 0x7ee9cb21 (stale) CRC-32 checksum value for chunk [0, 0x3d595): Mirror 1: 0x30b61926 Mirror 17: 0x30b61926 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d595: skipped 000483 1780332659.061953 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d595): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d595): Mirror 8: 0x7ee9cb21 (stale) Mirror 9: 0x7ee9cb21 (stale) Mirror 12: 0x7ee9cb21 (stale) Mirror 14: 0x7ee9cb21 (stale) Mirror 16: 0x7ee9cb21 (stale) CRC-32 checksum value for chunk [0, 0x3d595): Mirror 1: 0x30b61926 Mirror 17: 0x30b61926 Mirror 18: 0x30b61926 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d595: skipped 000484 1780332660.240992 read 0x020483 thru 0x02c917 (0xc495 bytes) 000485 1780332660.262617 mapread 0x010d39 thru 0x01e2ef (0xd5b7 bytes) 000487 1780332660.315682 mapwrite 0x00ef33 thru 0x017ff1 (0x90bf bytes) 000488 1780332660.409119 mapwrite 0x036344 thru 0x03ffff (0x9cbc bytes) 000489 1780332660.482981 trunc from 0x040000 to 0x029560 (0x16a9f bytes) 000490 1780332660.499851 read 0x00b11f thru 0x0167e9 (0xb6cb bytes) 000491 1780332660.501335 write 0x00d75f thru 0x015d83 (0x8625 bytes) 000492 1780332660.507344 fallocate 0x01e921 thru 0x02759c (0x8c7c bytes) 000493 1780332660.512516 punch from 0x01cc5d to 0x029560 (0xc904 bytes) 000494 1780332660.517587 write 0x00a8eb thru 0x012747 (0x7e5d bytes) 000495 1780332660.520379 mapwrite 0x00f667 thru 0x01c18b (0xcb25 bytes) 000497 1780332660.562695 trunc from 0x00134e to 0x008945 (0x75f8 bytes) 000498 1780332660.569756 trunc from 0x029561 to 0x00d1ed (0x1c373 bytes) 000499 1780332660.590411 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd1ee): Mirror 8: 0x1b47be13 (stale) Mirror 12: 0x1b47be13 (stale) Mirror 14: 0x1b47be13 (stale) Mirror 16: 0x1b47be13 (stale) Mirror 17: 0x451a89c2 (stale) Mirror 18: 0x451a89c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd1ee): Mirror 8: 0x1b47be13 (stale) Mirror 12: 0x1b47be13 (stale) Mirror 14: 0x1b47be13 (stale) Mirror 16: 0x1b47be13 (stale) Mirror 17: 0x451a89c2 (stale) Mirror 18: 0x451a89c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd1ee: skipped 000500 1780332661.024444 read 0x00c5a7 thru 0x00d1ed (0xc47 bytes) 000501 1780332661.037598 punch from 0x00ab70 to 0x00d1ed (0x267e bytes) 000502 1780332661.046568 trunc from 0x009159 to 0x00d1ed (0x4095 bytes) 000503 1780332661.053646 mapread 0x00c005 thru 0x00d1ed (0x11e9 bytes) 000504 1780332661.064883 trunc from 0x00d1ee to 0x034459 (0x2726c bytes) 000505 1780332661.087229 read 0x029e4a thru 0x02cfad (0x3164 bytes) 000506 1780332661.106427 trunc from 0x03445a to 0x02ba03 (0x8a56 bytes) 000507 1780332661.124069 mapwrite 0x010669 thru 0x01c07d (0xba15 bytes) 000508 1780332661.166315 fallocate 0x02fc3f thru 0x0377c2 (0x7b84 bytes) 000509 1780332661.175050 write 0x0224d1 thru 0x02380b (0x133b bytes) 000510 1780332661.177602 mapread 0x017aba thru 0x01cdef (0x5336 bytes) 000511 1780332661.179661 read 0x02b591 thru 0x0377c2 (0xc232 bytes) 000512 1780332661.189904 fallocate 0x034c38 thru 0x03e173 (0x953c bytes) 000513 1780332661.195506 read 0x01ee7b thru 0x024118 (0x529e bytes) 000515 1780332661.197543 punch from 0x0050c4 to 0x00780d (0x274a bytes) 000516 1780332661.208342 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x377c3): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x377c3): Mirror 8: 0x1629c372 (stale) Mirror 12: 0x1629c372 (stale) Mirror 14: 0x1629c372 (stale) Mirror 16: 0x1629c372 (stale) Mirror 17: 0x38683ff7 (stale) Mirror 18: 0x38683ff7 (stale) CRC-32 checksum value for chunk [0, 0x377c3): Mirror 1: 0xad70ef47 Mirror 19: 0xad70ef47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x377c3: skipped 000517 1780332662.226237 mapwrite 0x00e026 thru 0x0167f2 (0x87cd bytes) 000518 1780332662.352496 mapwrite 0x01196e thru 0x01a6e3 (0x8d76 bytes) 000519 1780332662.406763 trunc from 0x0377c3 to 0x0032f0 (0x344d2 bytes) 000520 1780332662.464868 MIRROR_RESYNC 8 mirrors 000521 1780332664.015466 trunc from 0x002c82 to 0x0032f0 (0x66f bytes) 000522 1780332664.046991 trunc from 0x0032f1 to 0x01c367 (0x19077 bytes) 000524 1780332664.068621 mapread 0x012186 thru 0x01a65a (0x84d5 bytes) 000525 1780332664.087823 mapread 0x00d37c thru 0x018ff3 (0xbc78 bytes) 000526 1780332664.089234 mapread 0x00472e thru 0x010ca8 (0xc57b bytes) 000527 1780332664.104510 mapwrite 0x00f75a thru 0x01d6ad (0xdf54 bytes) 000528 1780332664.154895 mapread 0x00154d thru 0x004962 (0x3416 bytes) 000529 1780332664.156929 mapwrite 0x0122d3 thru 0x0183ee (0x611c bytes) 000530 1780332664.184377 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d6ae): Mirror 8: 0x758b6ac7 (stale) Mirror 12: 0x758b6ac7 (stale) Mirror 14: 0x758b6ac7 (stale) Mirror 16: 0x758b6ac7 (stale) Mirror 18: 0x758b6ac7 (stale) Mirror 19: 0x758b6ac7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d6ae): Mirror 8: 0x758b6ac7 (stale) Mirror 12: 0x758b6ac7 (stale) Mirror 14: 0x758b6ac7 (stale) Mirror 16: 0x758b6ac7 (stale) Mirror 18: 0x758b6ac7 (stale) Mirror 19: 0x758b6ac7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d6ae: skipped 000531 1780332664.824027 trunc from 0x01d6ae to 0x03695c (0x192af bytes) 000532 1780332664.863981 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3695d): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3695d): Mirror 8: 0x22487ff7 (stale) Mirror 12: 0x22487ff7 (stale) Mirror 14: 0x22487ff7 (stale) Mirror 16: 0x22487ff7 (stale) Mirror 18: 0x22487ff7 (stale) Mirror 19: 0x22487ff7 (stale) CRC-32 checksum value for chunk [0, 0x3695d): Mirror 1: 0x41e0f85b Mirror 20: 0x41e0f85b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3695d: skipped 000533 1780332666.249837 read 0x017fe8 thru 0x01b600 (0x3619 bytes) 000534 1780332666.294018 mapread 0x034a69 thru 0x03695c (0x1ef4 bytes) 000535 1780332666.406983 mapread 0x00e928 thru 0x00ed25 (0x3fe bytes) 000536 1780332666.482754 mapwrite 0x02e13e thru 0x036bb6 (0x8a79 bytes) 000537 1780332666.653197 punch from 0x033b40 to 0x036bb6 (0x3077 bytes) 000539 1780332666.664432 mapread 0x000d26 thru 0x008e21 (0x80fc bytes) 000540 1780332666.678178 fallocate 0x00aba2 thru 0x00b9cb (0xe2a bytes) 000541 1780332666.692025 trunc from 0x018ada to 0x02201c (0x9543 bytes) 000542 1780332666.700911 mapwrite 0x034288 thru 0x03fc02 (0xb97b bytes) 000543 1780332666.765130 trunc from 0x01474f to 0x023aa1 (0xf353 bytes) 000544 1780332666.771415 write 0x023de2 thru 0x026639 (0x2858 bytes) 000545 1780332666.783476 mapwrite 0x02b43f thru 0x035446 (0xa008 bytes) 000546 1780332666.811084 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3fc03): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc03): Mirror 8: 0xdb8f0efb (stale) Mirror 12: 0xdb8f0efb (stale) Mirror 14: 0xdb8f0efb (stale) Mirror 16: 0xdb8f0efb (stale) Mirror 18: 0xdb8f0efb (stale) Mirror 19: 0xdb8f0efb (stale) Mirror 20: 0x3914b457 (stale) CRC-32 checksum value for chunk [0, 0x3fc03): Mirror 1: 0xaee9b153 Mirror 21: 0xaee9b153 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc03: skipped 000547 1780332667.730104 trunc from 0x023531 to 0x031f56 (0xea26 bytes) 000548 1780332667.779734 trunc from 0x02e248 to 0x03b531 (0xd2ea bytes) 000549 1780332667.786550 trunc from 0x03fc03 to 0x02249b (0x1d767 bytes) 000550 1780332667.814962 trunc from 0x02249c to 0x024bfc (0x2761 bytes) 000551 1780332667.838145 fallocate 0x00b263 thru 0x0121dc (0x6f7a bytes) 000552 1780332667.849085 MIRROR_RESYNC 9 mirrors 000554 1780332669.893963 mapwrite 0x0173e2 thru 0x01796b (0x58a bytes) 000555 1780332670.035809 fallocate 0x013c1a thru 0x02227e (0xe665 bytes) 000556 1780332670.055484 mapwrite 0x001e2c thru 0x00f53e (0xd713 bytes) 000557 1780332670.150819 mapwrite 0x0370ca thru 0x0386c5 (0x15fc bytes) 000559 1780332670.229942 write 0x02113e thru 0x02c2a6 (0xb169 bytes) 000560 1780332670.259848 trunc from 0x0386c6 to 0x0351db (0x34ea bytes) 000561 1780332670.314633 mapwrite 0x032cdc thru 0x033458 (0x77d bytes) 000562 1780332670.372601 MIRROR_RESYNC 9 mirrors 000564 1780332673.315247 write 0x0323b0 thru 0x032ca1 (0x8f2 bytes) 000565 1780332673.388729 mapread 0x01c19b thru 0x02a604 (0xe46a bytes) 000566 1780332673.428845 write 0x024e39 thru 0x027f2d (0x30f5 bytes) 000567 1780332673.430661 fallocate 0x00405d thru 0x00f476 (0xb41a bytes) 000568 1780332673.436975 trunc from 0x02709d to 0x02f051 (0x7fb5 bytes) 000569 1780332673.455221 trunc from 0x0351dc to 0x016440 (0x1ed9b bytes) 000570 1780332673.483730 trunc from 0x012950 to 0x016440 (0x3af1 bytes) 000571 1780332673.490489 write 0x0116e2 thru 0x01bcf7 (0xa616 bytes) 000572 1780332673.508547 write 0x0320a2 thru 0x038079 (0x5fd8 bytes) 000574 1780332673.512782 read 0x02dabd thru 0x038079 (0xa5bd bytes) 000575 1780332673.528020 MIRROR_RESYNC 9 mirrors 000576 1780332675.216095 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) 000577 1780332675.659452 read 0x018938 thru 0x01e4b1 (0x5b7a bytes) 000578 1780332675.670995 trunc from 0x036566 to 0x038079 (0x1b14 bytes) 000579 1780332675.758749 punch from 0x02d1a8 to 0x03467d (0x74d6 bytes) 000580 1780332675.766945 trunc from 0x03807a to 0x008021 (0x30058 bytes) 000581 1780332675.805722 punch from 0x001d1b to 0x008021 (0x6307 bytes) 000582 1780332675.812762 punch from 0x005540 to 0x008021 (0x2ae2 bytes) 000583 1780332675.819509 mapwrite 0x01e52a thru 0x02b97b (0xd452 bytes) 000584 1780332675.892373 fallocate 0x033b7a thru 0x0383c1 (0x4848 bytes) 000585 1780332675.899408 trunc from 0x029a30 to 0x02de2d (0x43fe bytes) 000586 1780332675.906653 trunc from 0x02a0d9 to 0x0342bb (0xa1e3 bytes) 000588 1780332675.913786 mapwrite 0x016df4 thru 0x021fa7 (0xb1b4 bytes) 000589 1780332675.944837 read 0x0133a1 thru 0x0216e1 (0xe341 bytes) 000590 1780332675.946741 trunc from 0x02dede to 0x037065 (0x9188 bytes) 000591 1780332675.952144 write 0x00d459 thru 0x00df91 (0xb39 bytes) 000592 1780332675.964214 mapwrite 0x0086e0 thru 0x00e18f (0x5ab0 bytes) 000593 1780332675.986615 trunc from 0x0383c2 to 0x00a7ec (0x2dbd5 bytes) 000594 1780332676.019685 MIRROR_RESYNC 8 mirrors 000595 1780332677.609621 trunc from 0x00a7ed to 0x006738 (0x40b4 bytes) 000596 1780332677.692661 fallocate 0x00e3ab thru 0x01c7f9 (0xe44f bytes) 000597 1780332677.699188 trunc from 0x006739 to 0x02b725 (0x24fed bytes) 000598 1780332677.738569 trunc from 0x02b726 to 0x013617 (0x1810e bytes) 000599 1780332677.755069 fallocate 0x00e1ee thru 0x012eeb (0x4cfe bytes) 000600 1780332677.768166 trunc from 0x013618 to 0x03ab3e (0x27527 bytes) 000601 1780332677.824098 read 0x01ad5d thru 0x01be1a (0x10be bytes) 000602 1780332677.834084 punch from 0x00b15b to 0x014c4f (0x9af5 bytes) 000603 1780332677.845593 trunc from 0x0020d1 to 0x003411 (0x1341 bytes) 000604 1780332677.856042 mapwrite 0x006150 thru 0x00c533 (0x63e4 bytes) 000605 1780332677.920996 read 0x012494 thru 0x0182c1 (0x5e2e bytes) 000606 1780332677.934674 mapread 0x02a264 thru 0x02ee68 (0x4c05 bytes) 000607 1780332677.936010 mapwrite 0x010720 thru 0x015e4f (0x5730 bytes) 000608 1780332677.965097 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ab3f): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab3f): Mirror 8: 0x671d738d (stale) Mirror 12: 0x671d738d (stale) Mirror 14: 0x671d738d (stale) Mirror 16: 0x671d738d (stale) Mirror 18: 0x671d738d (stale) Mirror 19: 0x671d738d (stale) Mirror 20: 0x671d738d (stale) CRC-32 checksum value for chunk [0, 0x3ab3f): Mirror 1: 0x106a64ba Mirror 21: 0x106a64ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab3f: skipped 000609 1780332679.213905 mapread 0x0039f8 thru 0x0120ef (0xe6f8 bytes) 000610 1780332679.245785 punch from 0x00977c to 0x014f5b (0xb7e0 bytes) 000611 1780332679.299690 write 0x02f71b thru 0x035d7c (0x6662 bytes) 000612 1780332679.314031 read 0x0146f8 thru 0x022cef (0xe5f8 bytes) 000614 1780332679.327647 fallocate 0x039fc3 thru 0x03ffff (0x603d bytes) 000615 1780332679.332740 mapread 0x03a82e thru 0x03ab3e (0x311 bytes) 000617 1780332679.338888 trunc from 0x03ab3f to 0x036e37 (0x3d07 bytes) 000618 1780332679.357493 fallocate 0x020973 thru 0x0234b1 (0x2b3f bytes) 000619 1780332679.362912 punch from 0x0180aa to 0x01fca0 (0x7bf7 bytes) 000620 1780332679.367627 trunc from 0x036e38 to 0x0101aa (0x26c8d bytes) 000621 1780332679.381643 write 0x013c4b thru 0x020979 (0xcd2f bytes) 000622 1780332679.383687 trunc from 0x02097a to 0x02638e (0x5a15 bytes) 000624 1780332679.413442 trunc from 0x00fdf1 to 0x01a174 (0xa384 bytes) 000625 1780332679.440669 mapwrite 0x00918f thru 0x009aa4 (0x916 bytes) 000626 1780332679.477346 read 0x022cf7 thru 0x02638e (0x3698 bytes) 000627 1780332679.501337 write 0x0054f3 thru 0x010944 (0xb452 bytes) 000628 1780332679.503556 trunc from 0x013c47 to 0x0161de (0x2598 bytes) 000629 1780332679.509541 read 0x00ee65 thru 0x015255 (0x63f1 bytes) 000630 1780332679.515726 fallocate 0x0101a4 thru 0x01be3a (0xbc97 bytes) 000631 1780332679.521138 fallocate 0x0060bd thru 0x009121 (0x3065 bytes) 000632 1780332679.528092 read 0x0194ae thru 0x022f2e (0x9a81 bytes) 000633 1780332679.529758 write 0x038747 thru 0x03ffff (0x78b9 bytes) 000634 1780332679.531945 mapread 0x00bb0a thru 0x013d96 (0x828d bytes) 000635 1780332679.534741 trunc from 0x009629 to 0x00c093 (0x2a6b bytes) 000636 1780332679.563672 fallocate 0x03038c thru 0x03ffff (0xfc74 bytes) 000637 1780332679.569193 read 0x03dbd6 thru 0x03ffff (0x242a bytes) 000639 1780332679.571116 mapwrite 0x03f498 thru 0x03ffff (0xb68 bytes) 000640 1780332679.588931 fallocate 0x036aba thru 0x03c994 (0x5edb bytes) 000641 1780332679.593686 mapread 0x037c21 thru 0x03ffff (0x83df bytes) 000642 1780332679.600497 punch from 0x020af5 to 0x02550c (0x4a18 bytes) 000643 1780332679.603847 fallocate 0x0158cb thru 0x016ddf (0x1515 bytes) 000644 1780332679.606861 trunc from 0x012ca2 to 0x0192af (0x660e bytes) 000645 1780332679.613197 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa6f02499 (stale) Mirror 14: 0xa6f02499 (stale) Mirror 16: 0xa6f02499 (stale) Mirror 18: 0xa6f02499 (stale) Mirror 19: 0xa6f02499 (stale) Mirror 20: 0xa6f02499 (stale) Mirror 21: 0x9229487e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa6f02499 (stale) Mirror 14: 0xa6f02499 (stale) Mirror 16: 0xa6f02499 (stale) Mirror 18: 0xa6f02499 (stale) Mirror 19: 0xa6f02499 (stale) Mirror 20: 0xa6f02499 (stale) Mirror 21: 0x9229487e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000646 1780332680.097192 MIRROR_RESYNC 8 mirrors 000647 1780332682.422817 mapread 0x0159a3 thru 0x01e60e (0x8c6c bytes) 000648 1780332682.443599 read 0x01c4fe thru 0x02a5a6 (0xe0a9 bytes) 000649 1780332682.461612 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) 000650 1780332682.841035 write 0x01928d thru 0x01eb7b (0x58ef bytes) 000651 1780332682.904072 mapwrite 0x02a227 thru 0x02c360 (0x213a bytes) 000652 1780332682.940294 mapwrite 0x02bb73 thru 0x03559b (0x9a29 bytes) 000653 1780332682.977828 write 0x029666 thru 0x037eaa (0xe845 bytes) 000654 1780332682.980931 mapread 0x01e3c8 thru 0x02d170 (0xeda9 bytes) 000655 1780332682.995086 fallocate 0x00145d thru 0x00c5ad (0xb151 bytes) 000656 1780332683.006243 mapread 0x037e5f thru 0x03ffff (0x81a1 bytes) 000657 1780332683.018037 trunc from 0x00b63e to 0x01038e (0x4d51 bytes) 000658 1780332683.025652 fallocate 0x015807 thru 0x021b3c (0xc336 bytes) 000659 1780332683.033202 trunc from 0x040000 to 0x024276 (0x1bd89 bytes) 000660 1780332683.062352 mapwrite 0x009143 thru 0x01381f (0xa6dd bytes) 000661 1780332683.125238 mapread 0x017a67 thru 0x01a3ea (0x2984 bytes) 000662 1780332683.130106 trunc from 0x024277 to 0x03aa8e (0x16818 bytes) 000663 1780332683.158868 trunc from 0x03aa8f to 0x03eb46 (0x40b8 bytes) 000665 1780332683.183381 mapwrite 0x009a2f thru 0x0112d8 (0x78aa bytes) 000666 1780332683.204871 trunc from 0x00dd7e to 0x019e6e (0xc0f1 bytes) 000667 1780332683.211894 fallocate 0x00735d thru 0x00e683 (0x7327 bytes) 000668 1780332683.218468 mapwrite 0x00c3f9 thru 0x01a18d (0xdd95 bytes) 000669 1780332683.264837 mapread 0x006383 thru 0x0150af (0xed2d bytes) 000670 1780332683.267961 mapread 0x009d18 thru 0x00ef71 (0x525a bytes) 000671 1780332683.269369 trunc from 0x03eb47 to 0x0255a3 (0x195a3 bytes) 000672 1780332683.289767 trunc from 0x0255a4 to 0x016728 (0xee7b bytes) 000673 1780332683.313666 write 0x016bbb thru 0x0233ac (0xc7f2 bytes) 000675 1780332683.325375 punch from 0x01195a to 0x014a05 (0x30ac bytes) 000676 1780332683.334730 write 0x004898 thru 0x00d629 (0x8d92 bytes) 000677 1780332683.336624 mapwrite 0x01367a thru 0x02090b (0xd292 bytes) 000679 1780332683.390319 trunc from 0x013825 to 0x01ee7f (0xb65b bytes) 000680 1780332683.403219 fallocate 0x00f27d thru 0x01ea6a (0xf7ee bytes) 000681 1780332683.410087 trunc from 0x0233ad to 0x004a9c (0x1e910 bytes) 000682 1780332683.435282 read 0x0049b9 thru 0x004a9c (0xe4 bytes) 000684 1780332683.439175 mapwrite 0x003e70 thru 0x00b830 (0x79c1 bytes) 000685 1780332683.511239 fallocate 0x038f68 thru 0x03f069 (0x6102 bytes) 000687 1780332683.523069 mapwrite 0x00bc62 thru 0x016dc3 (0xb162 bytes) 000688 1780332683.578099 punch from 0x002d1f to 0x011a3a (0xed1c bytes) 000690 1780332683.585951 punch from 0x008534 to 0x00e64f (0x611c bytes) 000691 1780332683.591801 punch from 0x00d600 to 0x016dc3 (0x97c4 bytes) 000692 1780332683.602581 MIRROR_RESYNC 7 mirrors 000693 1780332684.769060 read 0x0109d0 thru 0x016dc3 (0x63f4 bytes) 000694 1780332684.780076 mapread 0x0036a2 thru 0x009b52 (0x64b1 bytes) 000695 1780332684.802103 fallocate 0x0205af thru 0x0247b3 (0x4205 bytes) 000696 1780332684.854659 mapread 0x0011e0 thru 0x00cab7 (0xb8d8 bytes) 000697 1780332684.869676 punch from 0x008075 to 0x013dbe (0xbd4a bytes) 000698 1780332684.875038 read 0x010a6f thru 0x012924 (0x1eb6 bytes) 000699 1780332684.890553 read 0x016c7a thru 0x016dc3 (0x14a bytes) 000700 1780332684.891814 trunc from 0x01446b to 0x016dc3 (0x2959 bytes) 000701 1780332684.899347 write 0x034a67 thru 0x03ffff (0xb599 bytes) 000703 1780332684.902424 trunc from 0x03ff2b to 0x03ffff (0xd5 bytes) 000704 1780332684.916623 fallocate 0x00d32f thru 0x00f975 (0x2647 bytes) 000706 1780332684.926525 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 18: 0xe20eea22 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xe20eea22 (stale) Mirror 21: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x44d63aa1 Mirror 22: 0x44d63aa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000707 1780332686.061601 punch from 0x01aedd to 0x01f64b (0x476f bytes) 000708 1780332686.167154 fallocate 0x03a3bb thru 0x03ffff (0x5c45 bytes) 000709 1780332686.182676 trunc from 0x0363c5 to 0x03ffff (0x9c3b bytes) 000710 1780332686.190595 read 0x021e77 thru 0x0251ad (0x3337 bytes) 000711 1780332686.222143 write 0x02e924 thru 0x03e33b (0xfa18 bytes) 000712 1780332686.252866 read 0x02275b thru 0x02d9ca (0xb270 bytes) 000713 1780332686.282589 MIRROR_RESYNC 8 mirrors 000715 1780332689.171549 trunc from 0x040000 to 0x03e19d (0x1e62 bytes) 000716 1780332689.238192 write 0x010242 thru 0x01e843 (0xe602 bytes) 000718 1780332689.257426 punch from 0x00d903 to 0x01ce43 (0xf541 bytes) 000719 1780332689.274916 punch from 0x007f8b to 0x015640 (0xd6b6 bytes) 000721 1780332689.282037 punch from 0x014da7 to 0x01ad08 (0x5f62 bytes) 000723 1780332689.289163 write 0x009848 thru 0x0153ca (0xbb83 bytes) 000724 1780332689.318941 trunc from 0x03e19e to 0x03b7eb (0x29b2 bytes) 000725 1780332689.374082 fallocate 0x036f53 thru 0x037036 (0xe4 bytes) 000727 1780332689.381814 punch from 0x018a05 to 0x01948e (0xa8a bytes) 000728 1780332689.395099 trunc from 0x03b7ec to 0x00d19c (0x2e64f bytes) 000729 1780332689.422245 write 0x02e695 thru 0x03a36d (0xbcd9 bytes) 000730 1780332689.434932 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 14: 0xee7a6d8b (stale) Mirror 18: 0xee7a6d8b (stale) Mirror 19: 0xee7a6d8b (stale) Mirror 20: 0xee7a6d8b (stale) Mirror 21: 0xee7a6d8b (stale) Mirror 22: 0xee7a6d8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 14: 0xee7a6d8b (stale) Mirror 18: 0xee7a6d8b (stale) Mirror 19: 0xee7a6d8b (stale) Mirror 20: 0xee7a6d8b (stale) Mirror 21: 0xee7a6d8b (stale) Mirror 22: 0xee7a6d8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a36e: skipped 000731 1780332689.964985 MIRROR_RESYNC 7 mirrors 000733 1780332692.534184 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) 000734 1780332693.048913 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) 000735 1780332693.426010 trunc from 0x002e40 to 0x00925e (0x641f bytes) 000736 1780332693.475192 mapread 0x035690 thru 0x038002 (0x2973 bytes) 000737 1780332693.494460 fallocate 0x029db3 thru 0x03864a (0xe898 bytes) 000738 1780332693.499337 mapwrite 0x021cfe thru 0x0247fb (0x2afe bytes) 000739 1780332693.546780 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 18: 0x8e1c16b5 (stale) Mirror 20: 0x8e1c16b5 (stale) Mirror 21: 0x8e1c16b5 (stale) Mirror 22: 0x8e1c16b5 (stale) CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 1: 0x4f89333 Mirror 23: 0x4f89333 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a36e: skipped 000740 1780332694.642708 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 18: 0x8e1c16b5 (stale) Mirror 20: 0x8e1c16b5 (stale) Mirror 21: 0x8e1c16b5 (stale) Mirror 22: 0x8e1c16b5 (stale) CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 1: 0x4f89333 Mirror 23: 0x4f89333 Mirror 24: 0x4f89333 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a36e: skipped 000741 1780332695.766966 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 20: 0x8e1c16b5 (stale) Mirror 21: 0x8e1c16b5 (stale) Mirror 22: 0x8e1c16b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 20: 0x8e1c16b5 (stale) Mirror 21: 0x8e1c16b5 (stale) Mirror 22: 0x8e1c16b5 (stale) CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 1: 0x4f89333 Mirror 23: 0x4f89333 Mirror 24: 0x4f89333 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a36e: skipped 000743 1780332696.467025 read 0x02215f thru 0x03160e (0xf4b0 bytes) 000744 1780332696.480080 mapwrite 0x01e839 thru 0x01f522 (0xcea bytes) 000745 1780332696.633048 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 20: 0x8e1c16b5 (stale) Mirror 21: 0x8e1c16b5 (stale) Mirror 22: 0x8e1c16b5 (stale) Mirror 23: 0x4f89333 (stale) Mirror 24: 0x4f89333 (stale) CRC-32 checksum value for chunk [0, 0x3a36e): Mirror 1: 0xde55604e Mirror 25: 0xde55604e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a36e: skipped 000746 1780332697.629799 mapread 0x01de89 thru 0x02ac77 (0xcdef bytes) 000747 1780332697.676832 punch from 0x005d90 to 0x01593d (0xfbae bytes) 000749 1780332697.737052 trunc from 0x01378a to 0x014c16 (0x148d bytes) 000750 1780332697.741963 write 0x036cba thru 0x03ffff (0x9346 bytes) 000751 1780332697.759441 write 0x038258 thru 0x03ffff (0x7da8 bytes) 000752 1780332697.761033 mapwrite 0x003f32 thru 0x009f6e (0x603d bytes) 000753 1780332697.805991 fallocate 0x00bfb2 thru 0x0155b6 (0x9605 bytes) 000754 1780332697.813901 fallocate 0x001d79 thru 0x0063b2 (0x463a bytes) 000755 1780332697.818594 write 0x0391b8 thru 0x03c13b (0x2f84 bytes) 000756 1780332697.825069 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x5f08691b (stale) Mirror 21: 0x5f08691b (stale) Mirror 22: 0x5f08691b (stale) Mirror 23: 0x99f83b88 (stale) Mirror 24: 0x99f83b88 (stale) Mirror 25: 0xdc2eaeaf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfe6372a3 Mirror 26: 0xfe6372a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000757 1780332699.184333 MIRROR_RESYNC 8 mirrors 000758 1780332701.406698 trunc from 0x035520 to 0x03ffff (0xaae0 bytes) 000760 1780332701.462343 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0xfe6372a3 (stale) Mirror 21: 0xfe6372a3 (stale) Mirror 22: 0xfe6372a3 (stale) Mirror 23: 0xfe6372a3 (stale) Mirror 24: 0xfe6372a3 (stale) Mirror 25: 0xfe6372a3 (stale) Mirror 26: 0xfe6372a3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe59e998 Mirror 27: 0xe59e998 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000761 1780332702.553150 trunc from 0x040000 to 0x00d7a9 (0x32856 bytes) 000762 1780332702.608079 trunc from 0x0071e6 to 0x00cef2 (0x5d0d bytes) 000763 1780332702.616934 mapread 0x00b4da thru 0x00cb63 (0x168a bytes) 000764 1780332702.629139 write 0x0116ee thru 0x016d9c (0x56af bytes) 000765 1780332702.630892 trunc from 0x0136b7 to 0x016d9c (0x36e6 bytes) 000766 1780332702.645925 punch from 0x002199 to 0x00583b (0x36a3 bytes) 000767 1780332702.651691 fallocate 0x019970 thru 0x027675 (0xdd06 bytes) 000768 1780332702.657861 fallocate 0x03d654 thru 0x03ffff (0x29ac bytes) 000770 1780332702.666085 MIRROR_RESYNC 9 mirrors 000771 1780332704.942880 fallocate 0x02cd9c thru 0x037406 (0xa66b bytes) 000772 1780332705.022151 mapwrite 0x03e4ac thru 0x03f7bc (0x1311 bytes) 000773 1780332705.082898 trunc from 0x03f7bd to 0x03b9b9 (0x3e03 bytes) 000775 1780332705.107825 MIRROR_RESYNC 9 mirrors 000776 1780332707.656076 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) 000777 1780332708.124631 MIRROR_EXTEND to 9 mirrors 000778 1780332709.435812 read 0x0163cb thru 0x0223b5 (0xbfeb bytes) 000779 1780332709.445793 read 0x008ed7 thru 0x010478 (0x75a2 bytes) 000780 1780332709.456194 write 0x018f1f thru 0x022617 (0x96f9 bytes) 000781 1780332709.514243 fallocate 0x01294c thru 0x0140b8 (0x176d bytes) 000782 1780332709.524873 MIRROR_RESYNC 9 mirrors 000783 1780332712.299143 write 0x00c72d thru 0x00f069 (0x293d bytes) 000784 1780332712.357458 trunc from 0x03b9ba to 0x009aac (0x31f0d bytes) 000786 1780332712.380698 fallocate 0x01dfb5 thru 0x02d3fc (0xf448 bytes) 000787 1780332712.386420 trunc from 0x001179 to 0x009aac (0x8934 bytes) 000788 1780332712.398753 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x9aad): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9aad): Mirror 21: 0x8a84072c (stale) Mirror 22: 0x8a84072c (stale) Mirror 23: 0x8a84072c (stale) Mirror 24: 0x8a84072c (stale) Mirror 25: 0x8a84072c (stale) Mirror 26: 0x8a84072c (stale) Mirror 27: 0x8a84072c (stale) Mirror 28: 0x8a84072c (stale) CRC-32 checksum value for chunk [0, 0x9aad): Mirror 1: 0x225270a Mirror 29: 0x225270a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9aad: skipped 000789 1780332713.232658 fallocate 0x01c063 thru 0x0202a4 (0x4242 bytes) 000790 1780332713.289957 mapread 0x00c852 thru 0x016475 (0x9c24 bytes) 000791 1780332713.325044 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x202a5): Mirror 21: 0xe4e66e2a (stale) Mirror 22: 0xe4e66e2a (stale) Mirror 23: 0xe4e66e2a (stale) Mirror 24: 0xe4e66e2a (stale) Mirror 25: 0xe4e66e2a (stale) Mirror 27: 0xe4e66e2a (stale) Mirror 28: 0xe4e66e2a (stale) Mirror 29: 0xcdfced21 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x202a5): Mirror 21: 0xe4e66e2a (stale) Mirror 22: 0xe4e66e2a (stale) Mirror 23: 0xe4e66e2a (stale) Mirror 24: 0xe4e66e2a (stale) Mirror 25: 0xe4e66e2a (stale) Mirror 27: 0xe4e66e2a (stale) Mirror 28: 0xe4e66e2a (stale) Mirror 29: 0xcdfced21 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x202a5: skipped 000792 1780332713.867128 mapwrite 0x01fce2 thru 0x02fa38 (0xfd57 bytes) 000793 1780332713.979115 fallocate 0x02773b thru 0x02a669 (0x2f2f bytes) 000794 1780332713.985629 fallocate 0x02d0cc thru 0x03913b (0xc070 bytes) 000795 1780332713.991557 write 0x02348f thru 0x0298a8 (0x641a bytes) 000797 1780332713.994325 mapread 0x01de39 thru 0x02c0c2 (0xe28a bytes) 000798 1780332713.997093 mapwrite 0x01be71 thru 0x02a107 (0xe297 bytes) 000799 1780332714.038502 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fa39): Mirror 21: 0xe5f57bd (stale) Mirror 22: 0xe5f57bd (stale) Mirror 24: 0xe5f57bd (stale) Mirror 25: 0xe5f57bd (stale) Mirror 27: 0xe5f57bd (stale) Mirror 28: 0xe5f57bd (stale) Mirror 29: 0x5fe53a91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa39): Mirror 21: 0xe5f57bd (stale) Mirror 22: 0xe5f57bd (stale) Mirror 24: 0xe5f57bd (stale) Mirror 25: 0xe5f57bd (stale) Mirror 27: 0xe5f57bd (stale) Mirror 28: 0xe5f57bd (stale) Mirror 29: 0x5fe53a91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa39: skipped 000800 1780332714.619103 punch from 0x00ee51 to 0x01e6d6 (0xf886 bytes) 000801 1780332714.647037 fallocate 0x005661 thru 0x011a3b (0xc3db bytes) 000802 1780332714.658465 trunc from 0x01b77a to 0x02b087 (0xf90e bytes) 000803 1780332714.673618 mapwrite 0x027066 thru 0x028ca5 (0x1c40 bytes) 000804 1780332714.733284 mapwrite 0x000cc3 thru 0x00627c (0x55ba bytes) 000806 1780332714.778041 trunc from 0x02f50d to 0x02fa38 (0x52c bytes) 000808 1780332714.783049 mapread 0x00d0c7 thru 0x0129b5 (0x58ef bytes) 000809 1780332714.802009 fallocate 0x019887 thru 0x01a08f (0x809 bytes) 000811 1780332714.824626 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2fa39): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa39): Mirror 21: 0xe5f57bd (stale) Mirror 22: 0xe5f57bd (stale) Mirror 24: 0xe5f57bd (stale) Mirror 25: 0xe5f57bd (stale) Mirror 27: 0xe5f57bd (stale) Mirror 28: 0xe5f57bd (stale) Mirror 29: 0x5fe53a91 (stale) CRC-32 checksum value for chunk [0, 0x2fa39): Mirror 1: 0x6ff60b2d Mirror 30: 0x6ff60b2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa39: skipped 000812 1780332715.764471 write 0x037ba4 thru 0x03b9ad (0x3e0a bytes) 000813 1780332715.819952 punch from 0x036be5 to 0x03b9ad (0x4dc9 bytes) 000814 1780332715.859337 punch from 0x00fea7 to 0x01ae3d (0xaf97 bytes) 000816 1780332715.878597 trunc from 0x01582f to 0x016eb0 (0x1682 bytes) 000817 1780332715.884168 write 0x03edfc thru 0x03ffff (0x1204 bytes) 000819 1780332715.885718 mapread 0x03f103 thru 0x03ffff (0xefd bytes) 000820 1780332715.887334 fallocate 0x014e75 thru 0x016e36 (0x1fc2 bytes) 000821 1780332715.895010 trunc from 0x01a967 to 0x024899 (0x9f33 bytes) 000823 1780332715.904040 mapread 0x03839b thru 0x03ffff (0x7c65 bytes) 000824 1780332715.930013 trunc from 0x02ba65 to 0x02c5e0 (0xb7c bytes) 000825 1780332715.936137 fallocate 0x03e3f1 thru 0x03ffff (0x1c0f bytes) 000826 1780332715.948637 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x49251a60 (stale) Mirror 22: 0x49251a60 (stale) Mirror 24: 0x49251a60 (stale) Mirror 25: 0x49251a60 (stale) Mirror 27: 0x49251a60 (stale) Mirror 28: 0x49251a60 (stale) Mirror 29: 0xe20eea22 (stale) Mirror 30: 0xbd7e0173 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf87ee5d4 Mirror 31: 0xf87ee5d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000827 1780332717.337172 read 0x022972 thru 0x02e24f (0xb8de bytes) 000828 1780332717.358896 trunc from 0x00a410 to 0x015bd5 (0xb7c6 bytes) 000829 1780332717.415646 trunc from 0x003e28 to 0x009b12 (0x5ceb bytes) 000830 1780332717.421050 fallocate 0x03c4e5 thru 0x03ffff (0x3b1b bytes) 000833 1780332717.429886 trunc from 0x001aa3 to 0x00b18a (0x96e8 bytes) 000834 1780332717.436640 trunc from 0x0147db to 0x018606 (0x3e2c bytes) 000836 1780332717.442364 read 0x03282f thru 0x0392b5 (0x6a87 bytes) 000837 1780332717.458694 mapwrite 0x03c267 thru 0x03ffb8 (0x3d52 bytes) 000838 1780332717.474983 fallocate 0x01871d thru 0x01dcd4 (0x55b8 bytes) 000840 1780332717.481952 write 0x02ce26 thru 0x03957c (0xc757 bytes) 000841 1780332717.488977 write 0x023bf9 thru 0x0256cc (0x1ad4 bytes) 000842 1780332717.494965 trunc from 0x040000 to 0x021a23 (0x1e5dc bytes) 000844 1780332717.525943 mapread 0x020f41 thru 0x02198e (0xa4e bytes) 000845 1780332717.527717 trunc from 0x013397 to 0x013f4b (0xbb5 bytes) 000846 1780332717.533021 fallocate 0x02671e thru 0x036381 (0xfc64 bytes) 000848 1780332717.539454 fallocate 0x01a5ba thru 0x0270fe (0xcb45 bytes) 000849 1780332717.544476 mapread 0x00d4be thru 0x0113a3 (0x3ee6 bytes) 000850 1780332717.545836 punch from 0x00cb58 to 0x01cb10 (0xffb9 bytes) 000851 1780332717.552849 write 0x00ae2c thru 0x014b0b (0x9ce0 bytes) 000852 1780332717.559176 trunc from 0x0270ff to 0x014ef9 (0x12205 bytes) 000853 1780332717.576329 write 0x03e11a thru 0x03ffff (0x1ee6 bytes) 000854 1780332717.577321 trunc from 0x040000 to 0x01f193 (0x20e6c bytes) 000855 1780332717.599475 write 0x01a2fe thru 0x027d1d (0xda20 bytes) 000856 1780332717.607490 mapread 0x003ecb thru 0x00d280 (0x93b6 bytes) 000857 1780332717.608818 fallocate 0x00d1f6 thru 0x0179f2 (0xa7fd bytes) 000858 1780332717.619709 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x27d1e): Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27d1e): Mirror 21: 0xb5e52403 (stale) Mirror 22: 0xb5e52403 (stale) Mirror 24: 0xb5e52403 (stale) Mirror 25: 0xb5e52403 (stale) Mirror 27: 0xb5e52403 (stale) Mirror 28: 0xb5e52403 (stale) Mirror 29: 0xeec6ec42 (stale) Mirror 30: 0x46475bbc (stale) Mirror 31: 0x46475bbc (stale) CRC-32 checksum value for chunk [0, 0x27d1e): Mirror 1: 0x131e3415 Mirror 32: 0x131e3415 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27d1e: skipped 000859 1780332718.543873 punch from 0x022d25 to 0x027d1d (0x4ff9 bytes) 000860 1780332718.594312 mapwrite 0x03791b thru 0x03ffff (0x86e5 bytes) 000861 1780332718.688030 mapwrite 0x013499 thru 0x01781c (0x4384 bytes) 000862 1780332718.722193 write 0x024c1b thru 0x025048 (0x42e bytes) 000863 1780332718.732728 write 0x0086aa thru 0x018420 (0xfd77 bytes) 000865 1780332718.742844 write 0x02672f thru 0x028057 (0x1929 bytes) 000866 1780332718.745019 fallocate 0x02b11f thru 0x035ac0 (0xa9a2 bytes) 000868 1780332718.753594 punch from 0x000b46 to 0x00d740 (0xcbfb bytes) 000869 1780332718.770977 trunc from 0x040000 to 0x008e1e (0x371e1 bytes) 000870 1780332718.792716 punch from 0x008a68 to 0x008e1e (0x3b7 bytes) 000871 1780332718.800927 trunc from 0x0023f4 to 0x008e1e (0x6a2b bytes) 000872 1780332718.806974 fallocate 0x0102f7 thru 0x01157f (0x1289 bytes) 000873 1780332718.812597 trunc from 0x007633 to 0x008de9 (0x17b7 bytes) 000874 1780332718.817107 read 0x008f46 thru 0x00b2f0 (0x23ab bytes) 000875 1780332718.825007 fallocate 0x026d98 thru 0x02ec26 (0x7e8f bytes) 000876 1780332718.831545 write 0x02c7fd thru 0x038347 (0xbb4b bytes) 000877 1780332718.833563 mapread 0x01888b thru 0x0278d1 (0xf047 bytes) 000878 1780332718.853833 mapread 0x028fc8 thru 0x02afd1 (0x200a bytes) 000879 1780332718.861412 write 0x00933e thru 0x0122a0 (0x8f63 bytes) 000880 1780332718.868635 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38348): Mirror 22: 0x69a4fbf9 (stale) Mirror 24: 0x69a4fbf9 (stale) Mirror 25: 0x69a4fbf9 (stale) Mirror 27: 0x69a4fbf9 (stale) Mirror 28: 0x69a4fbf9 (stale) Mirror 29: 0x325881aa (stale) Mirror 30: 0xe660c48c (stale) Mirror 31: 0xa7b86731 (stale) Mirror 32: 0x93603207 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38348): Mirror 22: 0x69a4fbf9 (stale) Mirror 24: 0x69a4fbf9 (stale) Mirror 25: 0x69a4fbf9 (stale) Mirror 27: 0x69a4fbf9 (stale) Mirror 28: 0x69a4fbf9 (stale) Mirror 29: 0x325881aa (stale) Mirror 30: 0xe660c48c (stale) Mirror 31: 0xa7b86731 (stale) Mirror 32: 0x93603207 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38348: skipped 000881 1780332719.500243 read 0x005f71 thru 0x00d923 (0x79b3 bytes) 000882 1780332719.516500 mapread 0x000049 thru 0x00a01b (0x9fd3 bytes) 000883 1780332719.544028 trunc from 0x038348 to 0x03a6ac (0x2365 bytes) 000884 1780332719.567696 read 0x035806 thru 0x0364c5 (0xcc0 bytes) 000885 1780332719.581307 fallocate 0x03fcc9 thru 0x03ffff (0x337 bytes) 000886 1780332719.586286 mapwrite 0x007563 thru 0x00ae87 (0x3925 bytes) 000887 1780332719.603597 trunc from 0x03658f to 0x03a6ac (0x411e bytes) 000888 1780332719.608945 mapread 0x008dc3 thru 0x012d75 (0x9fb3 bytes) 000889 1780332719.611230 read 0x01ec07 thru 0x01f78b (0xb85 bytes) 000890 1780332719.612312 write 0x00e792 thru 0x01cb22 (0xe391 bytes) 000891 1780332719.614116 mapread 0x010950 thru 0x012a3f (0x20f0 bytes) 000892 1780332719.615869 fallocate 0x011f08 thru 0x01cdf7 (0xaef0 bytes) 000894 1780332719.623041 trunc from 0x03a6ad to 0x00df3d (0x2c76f bytes) 000895 1780332719.640848 fallocate 0x0388aa thru 0x03ffff (0x7756 bytes) 000896 1780332719.647007 trunc from 0x040000 to 0x0044ba (0x3bb45 bytes) 000897 1780332719.660811 fallocate 0x01f21d thru 0x026a27 (0x780b bytes) 000898 1780332719.666864 mapread 0x002ede thru 0x0044ba (0x15dd bytes) 000899 1780332719.667837 trunc from 0x0001b3 to 0x00223c (0x208a bytes) 000900 1780332719.674722 mapwrite 0x005984 thru 0x00905e (0x36db bytes) 000902 1780332719.717308 fallocate 0x023b27 thru 0x02491f (0xdf9 bytes) 000903 1780332719.728330 mapwrite 0x004c8c thru 0x010b82 (0xbef7 bytes) 000904 1780332719.776951 write 0x02fa88 thru 0x0307ff (0xd78 bytes) 000905 1780332719.778660 mapread 0x02bb0b thru 0x030363 (0x4859 bytes) 000907 1780332719.789514 punch from 0x00934a to 0x00c81c (0x34d3 bytes) 000908 1780332719.795611 fallocate 0x02ca43 thru 0x033684 (0x6c42 bytes) 000909 1780332719.806029 MIRROR_RESYNC 10 mirrors 000910 1780332722.019104 read 0x009ae8 thru 0x00ead6 (0x4fef bytes) 000911 1780332722.034521 MIRROR_EXTEND to 11 mirrors 000912 1780332723.064189 write 0x00cd31 thru 0x0101fe (0x34ce bytes) 000913 1780332723.155997 mapwrite 0x01f195 thru 0x0214fa (0x2366 bytes) 000914 1780332723.209853 write 0x03576b thru 0x039e1d (0x46b3 bytes) 000915 1780332723.215342 mapwrite 0x010800 thru 0x01328d (0x2a8e bytes) 000916 1780332723.255423 mapread 0x0089aa thru 0x015184 (0xc7db bytes) 000917 1780332723.279308 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x39e1e): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e1e): Mirror 22: 0xdebd4485 (stale) Mirror 24: 0xdebd4485 (stale) Mirror 25: 0xdebd4485 (stale) Mirror 27: 0xdebd4485 (stale) Mirror 28: 0xdebd4485 (stale) Mirror 29: 0xdebd4485 (stale) Mirror 30: 0xdebd4485 (stale) Mirror 31: 0xdebd4485 (stale) Mirror 32: 0xdebd4485 (stale) Mirror 33: 0xdebd4485 (stale) CRC-32 checksum value for chunk [0, 0x39e1e): Mirror 1: 0xaf0f6c33 Mirror 34: 0xaf0f6c33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e1e: skipped 000918 1780332724.550100 fallocate 0x00aebf thru 0x00cc06 (0x1d48 bytes) 000919 1780332724.612076 mapwrite 0x031db5 thru 0x033481 (0x16cd bytes) 000920 1780332724.661086 fallocate 0x00e0d3 thru 0x01cdfa (0xed28 bytes) 000921 1780332724.667456 punch from 0x03595f to 0x039e1d (0x44bf bytes) 000922 1780332724.674788 punch from 0x033476 to 0x03779f (0x432a bytes) 000923 1780332724.680501 mapwrite 0x000b2f thru 0x00728d (0x675f bytes) 000924 1780332724.712475 mapread 0x01769e thru 0x01f793 (0x80f6 bytes) 000927 1780332724.728919 trunc from 0x039e1e to 0x03bc2d (0x1e10 bytes) 000928 1780332724.782013 fallocate 0x018360 thru 0x019ff3 (0x1c94 bytes) 000929 1780332724.787091 mapread 0x012ed8 thru 0x019f9c (0x70c5 bytes) 000930 1780332724.789166 trunc from 0x03bc2e to 0x03fa34 (0x3e07 bytes) 000931 1780332724.811454 mapread 0x02e135 thru 0x037350 (0x921c bytes) 000932 1780332724.821339 mapread 0x028f41 thru 0x02c4ff (0x35bf bytes) 000934 1780332724.834607 punch from 0x002e99 to 0x0105ee (0xd756 bytes) 000935 1780332724.840906 trunc from 0x03fa35 to 0x033b99 (0xbe9b bytes) 000936 1780332724.855151 punch from 0x030d13 to 0x033b99 (0x2e87 bytes) 000937 1780332724.860837 fallocate 0x0393d0 thru 0x03ffff (0x6c30 bytes) 000938 1780332724.869150 fallocate 0x01f9b9 thru 0x028164 (0x87ac bytes) 000939 1780332724.874707 trunc from 0x01622c to 0x01bbe4 (0x59b9 bytes) 000940 1780332724.880367 mapwrite 0x0122d2 thru 0x013c86 (0x19b5 bytes) 000941 1780332724.899950 write 0x005cd3 thru 0x007780 (0x1aae bytes) 000942 1780332724.923235 read 0x00f3c5 thru 0x00fdb3 (0x9ef bytes) 000943 1780332724.923902 write 0x0067d5 thru 0x014f95 (0xe7c1 bytes) 000944 1780332724.928742 trunc from 0x022a41 to 0x02965c (0x6c1c bytes) 000945 1780332724.935327 punch from 0x017fee to 0x025c34 (0xdc47 bytes) 000946 1780332724.942677 write 0x03356b thru 0x03ffff (0xca95 bytes) 000947 1780332724.959560 mapwrite 0x001301 thru 0x00f193 (0xde93 bytes) 000948 1780332724.989950 mapwrite 0x03045f thru 0x03745f (0x7001 bytes) 000950 1780332725.003354 trunc from 0x021689 to 0x0273f1 (0x5d69 bytes) 000951 1780332725.019108 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xd13ccca2 (stale) Mirror 25: 0xd13ccca2 (stale) Mirror 27: 0xd13ccca2 (stale) Mirror 28: 0xd13ccca2 (stale) Mirror 29: 0xd13ccca2 (stale) Mirror 30: 0xd13ccca2 (stale) Mirror 31: 0xd13ccca2 (stale) Mirror 32: 0xd13ccca2 (stale) Mirror 33: 0xd13ccca2 (stale) Mirror 34: 0xc5edc5da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xd13ccca2 (stale) Mirror 25: 0xd13ccca2 (stale) Mirror 27: 0xd13ccca2 (stale) Mirror 28: 0xd13ccca2 (stale) Mirror 29: 0xd13ccca2 (stale) Mirror 30: 0xd13ccca2 (stale) Mirror 31: 0xd13ccca2 (stale) Mirror 32: 0xd13ccca2 (stale) Mirror 33: 0xd13ccca2 (stale) Mirror 34: 0xc5edc5da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000952 1780332725.579920 mapread 0x01d338 thru 0x0291c5 (0xbe8e bytes) 000953 1780332725.617816 trunc from 0x01120f to 0x0174b9 (0x62ab bytes) 000954 1780332725.627375 trunc from 0x040000 to 0x014bfb (0x2b404 bytes) 000955 1780332725.651917 fallocate 0x03c4de thru 0x03e635 (0x2158 bytes) 000956 1780332725.657224 trunc from 0x006bae to 0x014bfb (0xe04e bytes) 000957 1780332725.663857 mapwrite 0x03f6ce thru 0x03ffff (0x932 bytes) 000958 1780332725.699702 trunc from 0x035777 to 0x039b7b (0x4405 bytes) 000959 1780332725.705171 mapwrite 0x000d22 thru 0x008485 (0x7764 bytes) 000960 1780332725.733422 read 0x03e70d thru 0x03ffff (0x18f3 bytes) 000962 1780332725.735421 read 0x01c87b thru 0x029988 (0xd10e bytes) 000965 1780332725.764381 read 0x009779 thru 0x01132f (0x7bb7 bytes) 000966 1780332725.769061 punch from 0x039b13 to 0x03d2fe (0x37ec bytes) 000967 1780332725.777215 trunc from 0x039045 to 0x03ffff (0x6fbb bytes) 000968 1780332725.784154 punch from 0x023ebe to 0x02989e (0x59e1 bytes) 000969 1780332725.790439 read 0x015075 thru 0x019508 (0x4494 bytes) 000970 1780332725.791462 mapread 0x006891 thru 0x007c3f (0x13af bytes) 000971 1780332725.798488 punch from 0x003cdc to 0x010f4d (0xd272 bytes) 000972 1780332725.810442 read 0x003bde thru 0x00db3a (0x9f5d bytes) 000973 1780332725.825493 mapread 0x03eee7 thru 0x03ffff (0x1119 bytes) 000974 1780332725.829526 trunc from 0x00a453 to 0x011d86 (0x7934 bytes) 000975 1780332725.840618 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1a9:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xd13ccca2 (stale) Mirror 25: 0xd13ccca2 (stale) Mirror 27: 0xd13ccca2 (stale) Mirror 28: 0xd13ccca2 (stale) Mirror 29: 0xd13ccca2 (stale) Mirror 31: 0xd13ccca2 (stale) Mirror 32: 0xd13ccca2 (stale) Mirror 33: 0xd13ccca2 (stale) Mirror 34: 0xc5edc5da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xd13ccca2 (stale) Mirror 25: 0xd13ccca2 (stale) Mirror 27: 0xd13ccca2 (stale) Mirror 28: 0xd13ccca2 (stale) Mirror 29: 0xd13ccca2 (stale) Mirror 31: 0xd13ccca2 (stale) Mirror 32: 0xd13ccca2 (stale) Mirror 33: 0xd13ccca2 (stale) Mirror 34: 0xc5edc5da (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000976 1780332726.541924 write 0x017044 thru 0x01bf4c (0x4f09 bytes) 000977 1780332726.558943 mapread 0x03e7ef thru 0x03ffff (0x1811 bytes) 000978 1780332726.569475 mapread 0x018d68 thru 0x028756 (0xf9ef bytes) 000979 1780332726.594694 fallocate 0x02c577 thru 0x02ff78 (0x3a02 bytes) 000980 1780332726.600350 trunc from 0x040000 to 0x00e1ae (0x31e51 bytes) 000981 1780332726.618830 mapread 0x004070 thru 0x005d0c (0x1c9d bytes) 000982 1780332726.632442 read 0x007e87 thru 0x00c05a (0x41d4 bytes) 000983 1780332726.633399 trunc from 0x00e1af to 0x01ab43 (0xc995 bytes) 000984 1780332726.677167 trunc from 0x01ab44 to 0x005d59 (0x14dea bytes) 000985 1780332726.701873 mapread 0x0040db thru 0x005d59 (0x1c7f bytes) 000987 1780332726.707092 trunc from 0x005d5a to 0x00e798 (0x8a3f bytes) 000988 1780332726.731714 mapread 0x0024a0 thru 0x004caf (0x2810 bytes) 000989 1780332726.732795 trunc from 0x00e799 to 0x00d2e5 (0x14b3 bytes) 000990 1780332726.751687 trunc from 0x00af0b to 0x00d2e5 (0x23db bytes) 000992 1780332726.759978 mapread 0x00c386 thru 0x00cb61 (0x7dc bytes) 000993 1780332726.770929 punch from 0x001a0d to 0x00d2e5 (0xb8d9 bytes) 000994 1780332726.777460 write 0x033ec2 thru 0x03dc11 (0x9d50 bytes) 000995 1780332726.780566 trunc from 0x01ec16 to 0x025039 (0x6424 bytes) 000996 1780332726.785812 mapread 0x02bda6 thru 0x036ddf (0xb03a bytes) 000997 1780332726.792451 trunc from 0x0148e8 to 0x0197ba (0x4ed3 bytes) 000998 1780332726.796429 punch from 0x03c28f to 0x03dc11 (0x1983 bytes) 000999 1780332726.813542 fallocate 0x027f5b thru 0x036c9d (0xed43 bytes) 001000 1780332726.820469 trunc from 0x03dc12 to 0x03471b (0x94f6 bytes) All operations completed A-OK! PASS 70a (134s) == sanity-flr test 71: check mirror extend parent fid ==== 12:52:12 (1780332732) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x20:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 12:52:21 (1780332741) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130494 s, 16.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778021 s, 13.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x305:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x305:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 12:52:29 (1780332749) Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3226848 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4916800 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8247680 .. Extending file size to 5028640 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5212096 .. Extending file size to 6132224 .. Extending file size to 6859616 .. Extending file size to 8420448 .. Extending file size to 8588064 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8644032 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9349248 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 60929 60931 60933 60934 60938 192.168.201.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre2 (opts:) 192.168.201.113@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:53:50 (1780332830) Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5095232 .. Extending file size to 2254592 .. failed Extending file size to 6610336 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7568544 .. Extending file size to 5737056 .. Extending file size to 6009888 .. Extending file size to 8194528 .. Extending file size to 7929024 .. Extending file size to 8896288 .. Extending file size to 8961120 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9059392 .. Extending file size to 8296320 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8544160 .. Extending file size to 9203264 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62315 62316 62317 62318 62321 done 192.168.201.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre2 (opts:) 192.168.201.113@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:55:12 (1780332912) Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2624 1268680 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1868 1269436 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18852 3588168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24804 3582216 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43656 7170384 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2624 1268680 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1868 1269436 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18852 3588168 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24804 3582216 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43656 7170384 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105606 s, 19.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229396 s, 9.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x208a99be (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:55:33 (1780332933) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144194 s, 14.5 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 12:55:42 (1780332942) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f6:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:55:51 (1780332951) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.215286 s, 9.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:56:03 (1780332963) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103682 s, 20.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x30:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x30:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x31:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:56:12 (1780332972) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182628 s, 11.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ff:0x0] } PASS 204c (38s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:56:50 (1780333010) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg113-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922499 s, 11.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x301:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x300:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31f:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:57:45 (1780333065) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg113-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x321:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186387 s, 11.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x321:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x321:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x302:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:58:43 (1780333123) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg113-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg113-server: Pool lustre.test_204f created Adding targets to pool oleg113-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg113-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216718 s, 9.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x322:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg113-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg113-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg113-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:00:01 (1780333201) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:00:10 (1780333210) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:00:19 (1780333219) oleg113-server: Pool lustre.test_206 created Adding targets to pool oleg113-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg113-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg113-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg113-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg113-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:00:49 (1780333249) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109026 s, 9.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964985 s, 10.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x48de5987 Mirror 3: 0x48de5987 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:01:03 (1780333263) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:01:07 (1780333267) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:01:12 (1780333272) Starting client: oleg113-client.virtnet: -o user_xattr,flock 192.168.201.113@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0545476 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xcc0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1fe0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1fe0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1fe0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xfb0000 Read this (4 bytes): BBBB 192.168.201.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg113-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:01:57 (1780333317) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0673641 s, 122 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:02:05 (1780333325) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729238 s, 14.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x217 lmm_fid: [0x200000405:0x217:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x217 lmm_fid: [0x200000405:0x217:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x30e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611671 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 13:02:07.000000000 -0400 Modify: 2026-06-01 13:02:07.000000000 -0400 Change: 2026-06-01 13:02:09.000000000 -0400 Birth: 2026-06-01 13:02:07.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:02:15 (1780333335) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659965 s, 15.9 MB/s oleg113-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg113-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg113-server: O/280000401/d12/32c: File not found by ext2_lookup PASS 210b (27s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:02:42 (1780333362) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.330094 s, 124 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0355104 s, 115 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 1 13:02 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 1 13:02 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:02:53 (1780333373) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:03:02 (1780333382) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1895 sec =========== 13:03:13 (1780333393) === sanity-flr: start cleanup 13:03:14 (1780333394) === === sanity-flr: finish cleanup 13:03:24 (1780333404) ===