-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 23:14:07 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg443-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 23:14:28 (1788837268) === oleg443-client.virtnet: executing check_config_client /mnt/lustre oleg443-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg443-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff927a45212000.idle_timeout=debug osc.lustre-OST0001-osc-ffff927a45212000.idle_timeout=debug disable quota as required oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:15:11 (1788837311) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8227] == sanity-flr test 0a: lfs mirror create with -N option == 23:15:13 (1788837313) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:15:31 (1788837331) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:15:35 (1788837335) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:15:39 (1788837339) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:16:09 (1788837369) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:16:13 (1788837373) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 23:16:17 (1788837377) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff927a45212000.stats=clear osc.lustre-OST0001-osc-ffff927a45212000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:16:26 (1788837386) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:16:40 (1788837400) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:16:50 (1788837410) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:16:54 (1788837414) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:17:03 (1788837423) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 23:17:14 (1788837434) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:17:22 (1788837442) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:17:30 (1788837450) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:17:39 (1788837459) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:17:51 (1788837471) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:17:59 (1788837479) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.79035 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840642 s, 12.5 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:18:10 (1788837490) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.467456 s, 22.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:18:18 (1788837498) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:18:29 (1788837509) Stopping /mnt/lustre-ost1 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927a45212000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927a45212000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg443-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927a45212000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927a45212000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg443-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:19:48 (1788837588) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10465 s, 20.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927a45212000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927a45212000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg443-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (50s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:20:38 (1788837638) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927a45212000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927a45212000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg443-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927a45212000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927a45212000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg443-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (89s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:22:07 (1788837727) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.07539 s, 12.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 23:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.004s user 0m0.001s sys 0m0.184s Stopping /mnt/lustre-ost1 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff927a45212000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff927a45212000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.150s user 0m0.000s sys 0m0.199s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg443-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg443-server oleg443-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff927a45212000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927a45212000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.383s user 0m0.000s sys 0m0.176s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg443-server: oleg443-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg443-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg443-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff927a45212000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff927a45212000.ost_server_uuid in FULL state after 0 sec PASS 33b (103s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:23:50 (1788837830) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:23:54 (1788837834) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:23:58 (1788837838) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 23:24:02 (1788837842) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 23:24:11 (1788837851) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.684858 s, 18.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:24:24 (1788837864) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352065 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285138 s, 36.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:24:32 (1788837872) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:24:40 (1788837880) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00636224 s, 161 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00146496 s, 699 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00182753 s, 560 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00165523 s, 619 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 23:24:52 (1788837892) create test files with size 15732786 .. 32468481c0a56fbda069235a55bee779 - 8e78cb8cea964cfba84dc375e3642834 - 1e1ac89da1340c6919e87ebf41bb7474 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg443-client.virtnet /mnt/lustre (opts:) Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 23:25:20 (1788837920) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:25:33 (1788837933) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:25:41 (1788837941) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107143 s, 9.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030673 s, 34.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 23:25:50 (1788837950) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.10693 s, 19.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.397526 s, 10.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242526 s, 8.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 23:26:09 (1788837969) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 23:26:13 (1788837973) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:26:17 (1788837977) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420658 s, 24.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 23:26:26 (1788837986) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.462823 s, 6.8 MB/s PASS 44a (13s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:26:40 (1788838000) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:26:50 (1788838010) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.327124 s, 32.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg443-client.virtnet /mnt/lustre (opts:) Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:27:02 (1788838022) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff927a50cbc800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.37827 s, 27.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg443-client.virtnet /mnt/lustre (opts:) Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff927a48bf9000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:27:20 (1788838040) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:28:03 (1788838083) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:28:42 (1788838122) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 23:28:46 (1788838126) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.275603 s, 11.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.194107 s, 16.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5be1dc33 (nosync,stale) Mirror 2: 0x5be1dc33 (nosync,stale) ** verify mirror 2 doesn't change original checksum: af115d490efc88a36281abb0dab7c8b3 - mirror 1 checksum: 1bde8c0292efe38b27372f9b637eb44d - mirror 2 checksum: af115d490efc88a36281abb0dab7c8b3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T23:28:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T23:28:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: af115d490efc88a36281abb0dab7c8b3 - mirror 1 checksum: 1bde8c0292efe38b27372f9b637eb44d - mirror 2 checksum: 1bde8c0292efe38b27372f9b637eb44d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T23:28:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T23:28:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:28:58 (1788838138) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602613 s, 17.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:29:08 (1788838148) Failing mds1 on oleg443-server Stopping /mnt/lustre-mds1 (opts:) on oleg443-server 23:29:21 (1788838161) shut down facet: mds1 facet_host: oleg443-server facet_failover_host: oleg443-server Failover mds1 to oleg443-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 23:29:54 (1788838194) targets are mounted 23:29:54 (1788838194) facet_failover done oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475917 s, 22.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:30:11 (1788838211) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.028913 s, 708 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.01286 s, 1.6 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:30:23 (1788838223) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0749296 s, 273 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0156343 s, 1.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:30:34 (1788838234) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0259871 s, 630 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:30:42 (1788838242) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0775466 s, 264 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:30:51 (1788838251) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0228995 s, 179 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0265615 s, 154 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:31:00 (1788838260) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:31:25 (1788838285) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:31:54 (1788838314) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:32:19 (1788838339) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:32:44 (1788838364) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 23:32:54 (1788838374) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57579 mirror split pid 57580 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57579 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57580 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 23:34:04 (1788838444) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 8447 skipping zero length zero range skipping zero length punch hole skipping zero length zero range 000004 1788838447.404428 mapwrite 0x021669 thru 0x030a2d (0xf3c5 bytes) 000006 1788838447.539640 read 0x02e2af thru 0x02ed9a (0xaec bytes) 000009 1788838447.560617 MIRROR_EXTEND to 2 mirrors 000010 1788838448.624435 write 0x039537 thru 0x039847 (0x311 bytes) 000011 1788838448.677470 write 0x017e98 thru 0x020444 (0x85ad bytes) 000012 1788838448.701878 mapread 0x01a477 thru 0x0288db (0xe465 bytes) 000014 1788838448.732742 mapread 0x02f961 thru 0x032079 (0x2719 bytes) fallocating to largest ever: 0x2ebdd 000016 1788838448.742780 fallocate 0x022aea thru 0x02ebdc (0xc0f3 bytes) 000017 1788838448.747473 punch from 0x00319b to 0x00ac8d (0x7af3 bytes) fallocating to largest ever: 0x3df94 000018 1788838448.752486 fallocate 0x03d1cb thru 0x03df93 (0xdc9 bytes) 000019 1788838448.763290 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000020 1788838449.026653 write 0x032483 thru 0x0336f8 (0x1276 bytes) 000021 1788838449.062556 trunc from 0x00a2e8 to 0x010c3b (0x6954 bytes) 000022 1788838449.086843 mapwrite 0x01993b thru 0x021634 (0x7cfa bytes) 000023 1788838449.147332 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x40000 000024 1788838450.141855 fallocate 0x03f95b thru 0x03ffff (0x6a5 bytes) 000025 1788838450.187484 mapwrite 0x03e7ee thru 0x03ffff (0x1812 bytes) 000026 1788838450.224328 trunc from 0x0023e7 to 0x00b88d (0x94a7 bytes) 000027 1788838450.230447 read 0x03c001 thru 0x03ffff (0x3fff bytes) 000028 1788838450.242852 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000031 1788838450.429325 mapread 0x0266e7 thru 0x03205c (0xb976 bytes) 000032 1788838450.497505 MIRROR_EXTEND to 2 mirrors 000035 1788838451.820191 mapread 0x030d95 thru 0x037987 (0x6bf3 bytes) 000037 1788838451.911877 fallocate 0x0282f8 thru 0x02a038 (0x1d41 bytes) 000039 1788838451.956401 read 0x00ba1f thru 0x00ddc0 (0x23a2 bytes) 000040 1788838451.964605 mapwrite 0x0016de thru 0x007b16 (0x6439 bytes) 000041 1788838452.007346 read 0x019428 thru 0x021936 (0x850f bytes) 000042 1788838452.048472 write 0x01f491 thru 0x02d295 (0xde05 bytes) 000043 1788838452.068935 trunc from 0x000c07 to 0x0034ca (0x28c4 bytes) 000044 1788838452.081705 fallocate 0x0294ac thru 0x0366b1 (0xd206 bytes) 000045 1788838452.091432 mapread 0x00c471 thru 0x0181bd (0xbd4d bytes) 000046 1788838452.100921 fallocate 0x00be15 thru 0x00e63e (0x282a bytes) 000047 1788838452.132013 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000049 1788838452.523011 read 0x031271 thru 0x03745c (0x61ec bytes) 000050 1788838452.538280 write 0x03e41d thru 0x03ffff (0x1be3 bytes) 000051 1788838452.561153 trunc from 0x02a567 to 0x02b55b (0xff5 bytes) 000052 1788838452.568381 mapread 0x01d4d9 thru 0x0245eb (0x7113 bytes) 000053 1788838452.630179 trunc from 0x040000 to 0x02870c (0x178f3 bytes) 000054 1788838452.664560 punch from 0x012385 to 0x017c6e (0x58ea bytes) 000056 1788838452.691835 MIRROR_EXTEND to 2 mirrors 000057 1788838453.642255 mapread 0x024e98 thru 0x02870c (0x3875 bytes) 000058 1788838453.683309 write 0x015a4e thru 0x01b254 (0x5807 bytes) 000059 1788838453.734672 punch from 0x001495 to 0x002b44 (0x16b0 bytes) 000060 1788838453.744790 mapread 0x00ad5a thru 0x016221 (0xb4c8 bytes) 000062 1788838453.771084 MIRROR_RESYNC 2 mirrors 000063 1788838454.537766 mapread 0x00e270 thru 0x01bc03 (0xd994 bytes) 000064 1788838454.579813 trunc from 0x02870d to 0x0032fd (0x2540f bytes) 000065 1788838454.634234 mapread 0x0008d6 thru 0x0032fd (0x2a28 bytes) 000066 1788838454.645003 punch from 0x002a54 to 0x0032fd (0x8aa bytes) 000067 1788838454.652317 trunc from 0x000a6f to 0x0032fd (0x288f bytes) 000068 1788838454.659298 mapwrite 0x009be9 thru 0x00b394 (0x17ac bytes) 000069 1788838454.714618 mapread 0x005e79 thru 0x0072a0 (0x1428 bytes) 000070 1788838454.715578 trunc from 0x00b395 to 0x03d916 (0x32582 bytes) 000071 1788838454.749245 trunc from 0x01b9f8 to 0x028412 (0xca1b bytes) 000072 1788838454.757549 trunc from 0x001ded to 0x00c553 (0xa767 bytes) 000073 1788838454.766209 read 0x01b058 thru 0x01b4e1 (0x48a bytes) 000074 1788838454.774561 read 0x02f9fd thru 0x03c7d0 (0xcdd4 bytes) 000075 1788838454.789075 trunc from 0x03d917 to 0x00a46e (0x334a8 bytes) 000076 1788838454.841652 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xa46f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa46f): Mirror 2: 0xa1c35c06 (stale) CRC-32 checksum value for chunk [0, 0xa46f): Mirror 1: 0x153e984f Mirror 3: 0x153e984f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa46f: skipped 000077 1788838455.904805 trunc from 0x00a46f to 0x0118ea (0x747c bytes) 000078 1788838455.995684 trunc from 0x006b92 to 0x00fabe (0x8f2d bytes) 000079 1788838456.026688 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x118eb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x118eb): Mirror 2: 0xfe9a9dd6 (stale) Mirror 3: 0xc1379c05 (stale) CRC-32 checksum value for chunk [0, 0x118eb): Mirror 1: 0xc1379c05 Mirror 4: 0xc1379c05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x118eb: skipped 000080 1788838457.325511 read 0x0117d9 thru 0x0118ea (0x112 bytes) 000081 1788838457.339406 trunc from 0x00f2be to 0x0118ea (0x262d bytes) 000082 1788838457.388092 write 0x03d95b thru 0x03ffff (0x26a5 bytes) 000083 1788838457.394214 read 0x01b59a thru 0x01e17d (0x2be4 bytes) 000084 1788838457.410841 trunc from 0x034dbd to 0x039927 (0x4b6b bytes) 000086 1788838457.420154 read 0x01b34f thru 0x02a8b0 (0xf562 bytes) 000087 1788838457.421426 read 0x01c48b thru 0x020025 (0x3b9b bytes) 000088 1788838457.422638 read 0x00cf8b thru 0x017358 (0xa3ce bytes) 000089 1788838457.423884 mapwrite 0x0123ce thru 0x01d349 (0xaf7c bytes) 000090 1788838457.445131 read 0x01c6a7 thru 0x024de0 (0x873a bytes) 000091 1788838457.445917 trunc from 0x040000 to 0x024cc3 (0x1b33c bytes) 000092 1788838457.471637 fallocate 0x033880 thru 0x03ffff (0xc780 bytes) 000093 1788838457.479399 punch from 0x0204c4 to 0x0209e4 (0x521 bytes) 000094 1788838457.485665 read 0x0227fb thru 0x024cc3 (0x24c9 bytes) 000095 1788838457.488178 read 0x024124 thru 0x024cc3 (0xba0 bytes) 000096 1788838457.490411 read 0x001da8 thru 0x0075c1 (0x581a bytes) 000097 1788838457.491445 punch from 0x0226cd to 0x024cc3 (0x25f7 bytes) 000098 1788838457.497798 trunc from 0x01cdd0 to 0x024cc3 (0x7ef4 bytes) 000099 1788838457.505149 read 0x01c14c thru 0x022bc2 (0x6a77 bytes) 000100 1788838457.517797 read 0x021609 thru 0x024cc3 (0x36bb bytes) 000101 1788838457.518657 mapread 0x011c03 thru 0x01f69d (0xda9b bytes) 000102 1788838457.521591 trunc from 0x024cc4 to 0x02ff61 (0xb29e bytes) 000103 1788838457.546887 MIRROR_RESYNC 4 mirrors 000104 1788838458.855519 trunc from 0x02ff62 to 0x03faa2 (0xfb41 bytes) 000105 1788838458.983559 trunc from 0x0042c3 to 0x010762 (0xc4a0 bytes) 000106 1788838458.992807 mapread 0x00812d thru 0x00a150 (0x2024 bytes) 000107 1788838459.038679 punch from 0x01113c to 0x01c601 (0xb4c6 bytes) 000108 1788838459.046618 read 0x00dbbe thru 0x0194b9 (0xb8fc bytes) 000109 1788838459.065192 trunc from 0x03cf5f to 0x03d228 (0x2ca bytes) 000110 1788838459.077966 MIRROR_RESYNC 4 mirrors 000111 1788838460.236006 fallocate 0x01b75a thru 0x02b2fc (0xfba3 bytes) 000112 1788838460.305563 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3faa3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3faa3): Mirror 2: 0xe0b5a8a8 (stale) Mirror 3: 0xe0b5a8a8 (stale) Mirror 4: 0xe0b5a8a8 (stale) CRC-32 checksum value for chunk [0, 0x3faa3): Mirror 1: 0xe0b5a8a8 Mirror 5: 0xe0b5a8a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3faa3: skipped 000113 1788838461.584133 read 0x029bec thru 0x032410 (0x8825 bytes) 000114 1788838461.626680 write 0x0392b1 thru 0x03ffff (0x6d4f bytes) 000115 1788838461.723677 mapwrite 0x03e684 thru 0x03ffff (0x197c bytes) 000117 1788838461.773127 read 0x00cbbd thru 0x0124a9 (0x58ed bytes) 000118 1788838461.794479 punch from 0x00d458 to 0x01aac7 (0xd670 bytes) 000119 1788838461.806516 trunc from 0x040000 to 0x031aa5 (0xe55a bytes) 000120 1788838461.849384 punch from 0x0312b8 to 0x031aa5 (0x7ee bytes) 000121 1788838461.854675 trunc from 0x022b9e to 0x0299ce (0x6e31 bytes) 000122 1788838461.860647 punch from 0x02506e to 0x02add8 (0x5d6b bytes) 000123 1788838461.869654 read 0x018bbb thru 0x02736d (0xe7b3 bytes) 000124 1788838461.903649 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31aa6): Mirror 2: 0x92867798 (stale) Mirror 4: 0x92867798 (stale) Mirror 5: 0x92867798 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31aa6): Mirror 2: 0x92867798 (stale) Mirror 4: 0x92867798 (stale) Mirror 5: 0x92867798 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31aa6: skipped 000126 1788838462.555645 trunc from 0x00261f to 0x00a270 (0x7c52 bytes) 000128 1788838462.564193 trunc from 0x031aa6 to 0x02c985 (0x5120 bytes) 000129 1788838462.591478 mapwrite 0x03d522 thru 0x03ffff (0x2ade bytes) 000130 1788838462.643135 mapwrite 0x03432d thru 0x038cac (0x4980 bytes) 000131 1788838462.676387 read 0x01a537 thru 0x021e29 (0x78f3 bytes) 000132 1788838462.697927 fallocate 0x03d297 thru 0x03ffff (0x2d69 bytes) 000133 1788838462.708026 punch from 0x006aa8 to 0x012af9 (0xc052 bytes) 000134 1788838462.713671 read 0x032cf1 thru 0x03e31b (0xb62b bytes) 000135 1788838462.717412 trunc from 0x00eae5 to 0x010aae (0x1fca bytes) 000136 1788838462.724613 punch from 0x0059b6 to 0x014ee0 (0xf52b bytes) 000137 1788838462.732338 punch from 0x031fba to 0x035e40 (0x3e87 bytes) 000138 1788838462.739604 mapread 0x016ac0 thru 0x01e51f (0x7a60 bytes) 000139 1788838462.754190 read 0x03acbe thru 0x03ffff (0x5342 bytes) 000140 1788838462.755103 mapwrite 0x03291b thru 0x035fe8 (0x36ce bytes) 000141 1788838462.776525 read 0x017bed thru 0x024a33 (0xce47 bytes) 000142 1788838462.777336 read 0x03f030 thru 0x03ffff (0xfd0 bytes) 000143 1788838462.778612 punch from 0x01e563 to 0x01fa04 (0x14a2 bytes) 000144 1788838462.789907 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x56f5a330 (stale) Mirror 4: 0x56f5a330 (stale) Mirror 5: 0x56f5a330 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b9e236d Mirror 6: 0x4b9e236d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000145 1788838464.297450 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x56f5a330 (stale) Mirror 5: 0x56f5a330 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x56f5a330 (stale) Mirror 5: 0x56f5a330 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b9e236d Mirror 6: 0x4b9e236d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000146 1788838465.112308 trunc from 0x03d2a4 to 0x03ffff (0x2d5c bytes) 000149 1788838465.171513 trunc from 0x040000 to 0x03b96a (0x4695 bytes) 000151 1788838465.223012 read 0x01870f thru 0x022eb0 (0xa7a2 bytes) 000152 1788838465.285681 MIRROR_RESYNC 4 mirrors 000153 1788838466.726175 write 0x00740f thru 0x00f1a1 (0x7d93 bytes) 000154 1788838466.770062 mapwrite 0x02592e thru 0x02a258 (0x492b bytes) 000155 1788838466.813555 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b96b): Mirror 5: 0x35e8cc5b (stale) Mirror 6: 0x35e8cc5b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b96b): Mirror 5: 0x35e8cc5b (stale) Mirror 6: 0x35e8cc5b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b96b: skipped 000156 1788838467.176521 trunc from 0x03b96b to 0x032935 (0x9035 bytes) 000157 1788838467.200463 trunc from 0x032936 to 0x00912e (0x29807 bytes) 000158 1788838467.232392 MIRROR_RESYNC 3 mirrors 000159 1788838468.127339 fallocate 0x01bcec thru 0x025696 (0x99ab bytes) 000160 1788838468.191922 mapwrite 0x032743 thru 0x03ff4d (0xd80b bytes) 000162 1788838468.343218 mapwrite 0x0160d2 thru 0x0209b0 (0xa8df bytes) 000163 1788838468.435003 MIRROR_RESYNC 3 mirrors 000164 1788838469.847353 fallocate 0x011e32 thru 0x0197bb (0x798a bytes) 000165 1788838469.895812 MIRROR_RESYNC 3 mirrors 000166 1788838471.145854 trunc from 0x025763 to 0x0287d1 (0x306f bytes) 000167 1788838471.197994 trunc from 0x03ff4e to 0x0325c3 (0xd98a bytes) 000168 1788838471.241556 mapread 0x029dad thru 0x0325c3 (0x8817 bytes) 000170 1788838471.275725 mapwrite 0x0368ec thru 0x03e773 (0x7e88 bytes) 000171 1788838471.358927 fallocate 0x00bc6c thru 0x01aa14 (0xeda9 bytes) 000172 1788838471.371790 fallocate 0x0324ca thru 0x033222 (0xd59 bytes) 000173 1788838471.381386 fallocate 0x00c187 thru 0x0155e2 (0x945c bytes) 000174 1788838471.392783 fallocate 0x014953 thru 0x01db2c (0x91da bytes) 000175 1788838471.408013 mapwrite 0x0016fe thru 0x00feda (0xe7dd bytes) 000176 1788838471.449960 read 0x01c167 thru 0x020fde (0x4e78 bytes) 000177 1788838471.460401 mapread 0x018d8a thru 0x0210b2 (0x8329 bytes) 000178 1788838471.503479 mapwrite 0x013404 thru 0x01a36c (0x6f69 bytes) 000179 1788838471.544378 mapwrite 0x035dca thru 0x03ffff (0xa236 bytes) 000180 1788838471.642654 punch from 0x0170c0 to 0x026069 (0xefaa bytes) 000181 1788838471.654150 write 0x03c530 thru 0x03ffff (0x3ad0 bytes) 000183 1788838471.655799 fallocate 0x02063d thru 0x02ae9f (0xa863 bytes) 000184 1788838471.667291 fallocate 0x00f08b thru 0x01cde3 (0xdd59 bytes) 000185 1788838471.680686 trunc from 0x040000 to 0x021bfa (0x1e405 bytes) 000186 1788838471.718919 fallocate 0x00ff52 thru 0x0128fa (0x29a9 bytes) 000187 1788838471.725500 write 0x008ff9 thru 0x018e98 (0xfea0 bytes) 000189 1788838471.759210 mapwrite 0x037923 thru 0x0387d1 (0xeaf bytes) 000190 1788838471.830243 punch from 0x0038ce to 0x01055f (0xcc92 bytes) 000192 1788838471.837146 read 0x017902 thru 0x020fee (0x96ed bytes) 000193 1788838471.865090 trunc from 0x0004ce to 0x009e62 (0x9995 bytes) 000195 1788838471.879089 punch from 0x01d540 to 0x01e8c8 (0x1389 bytes) 000196 1788838471.884479 write 0x014630 thru 0x01614e (0x1b1f bytes) 000197 1788838471.887796 trunc from 0x0387d2 to 0x01be57 (0x1c97a bytes) 000198 1788838471.924600 mapwrite 0x01ac05 thru 0x0253fe (0xa7fa bytes) 000199 1788838472.101933 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x253ff): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x253ff): Mirror 5: 0x80ce6d40 (stale) Mirror 6: 0x80ce6d40 (stale) CRC-32 checksum value for chunk [0, 0x253ff): Mirror 1: 0xf96cbfbb Mirror 7: 0xf96cbfbb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x253ff: skipped 000200 1788838473.422972 punch from 0x0227d9 to 0x0253fe (0x2c26 bytes) 000201 1788838473.542254 trunc from 0x0253ff to 0x03b492 (0x16094 bytes) 000202 1788838473.608314 trunc from 0x03b493 to 0x00ccf8 (0x2e79a bytes) 000204 1788838473.658618 trunc from 0x00ccf9 to 0x03d5ff (0x30907 bytes) 000205 1788838473.696182 trunc from 0x03d600 to 0x0148ad (0x28d52 bytes) 000206 1788838473.729856 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x148ae): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x148ae): Mirror 5: 0x3502a707 (stale) Mirror 6: 0x3502a707 (stale) Mirror 7: 0xa54a66df (stale) CRC-32 checksum value for chunk [0, 0x148ae): Mirror 1: 0x78cd0bd8 Mirror 8: 0x78cd0bd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x148ae: skipped 000207 1788838474.649349 mapwrite 0x0209a0 thru 0x02531d (0x497e bytes) 000208 1788838474.786883 fallocate 0x01037d thru 0x018b2b (0x87af bytes) 000209 1788838474.801763 fallocate 0x012526 thru 0x0211ac (0xec87 bytes) 000211 1788838474.821750 mapwrite 0x00bb1d thru 0x00dde2 (0x22c6 bytes) 000212 1788838474.856022 write 0x00bf2c thru 0x013006 (0x70db bytes) 000213 1788838474.858399 read 0x00185e thru 0x00cf7f (0xb722 bytes) 000214 1788838474.866250 trunc from 0x021e1f to 0x02441a (0x25fc bytes) 000215 1788838474.874034 trunc from 0x02531e to 0x02e1f5 (0x8ed8 bytes) 000216 1788838474.918256 mapread 0x006caf thru 0x01428b (0xd5dd bytes) 000218 1788838474.937544 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e1f6): Mirror 5: 0xb5bceb0a (stale) Mirror 6: 0xb5bceb0a (stale) Mirror 7: 0x88e9484f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e1f6): Mirror 5: 0xb5bceb0a (stale) Mirror 6: 0xb5bceb0a (stale) Mirror 7: 0x88e9484f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e1f6: skipped 000219 1788838475.629807 trunc from 0x02e1f6 to 0x003ac6 (0x2a72f bytes) 000220 1788838475.684622 trunc from 0x002583 to 0x003ac6 (0x1544 bytes) 000221 1788838475.701748 mapwrite 0x017432 thru 0x01968d (0x225c bytes) 000223 1788838475.828766 trunc from 0x01968e to 0x011d0c (0x7981 bytes) 000224 1788838475.855283 punch from 0x00b220 to 0x011d0c (0x6aed bytes) 000225 1788838475.870368 mapread 0x006fcc thru 0x010375 (0x93aa bytes) 000226 1788838475.892123 write 0x01c30c thru 0x020b14 (0x4809 bytes) 000227 1788838475.898069 read 0x01f702 thru 0x020b14 (0x1413 bytes) 000228 1788838475.899101 write 0x012909 thru 0x0214af (0xeba7 bytes) 000229 1788838475.911627 MIRROR_RESYNC 4 mirrors 000230 1788838476.843513 trunc from 0x0214b0 to 0x00e7b3 (0x12cfc bytes) 000231 1788838476.890799 write 0x0356e1 thru 0x0370a0 (0x19c0 bytes) 000232 1788838476.893000 punch from 0x027072 to 0x03542c (0xe3bb bytes) 000233 1788838476.909870 punch from 0x02abe0 to 0x034a82 (0x9ea3 bytes) 000234 1788838476.916036 write 0x0345f4 thru 0x03fa60 (0xb46d bytes) 000235 1788838476.922515 fallocate 0x004525 thru 0x00c1f1 (0x7ccd bytes) 000237 1788838476.929517 mapwrite 0x031369 thru 0x03fc73 (0xe90b bytes) 000238 1788838476.994214 trunc from 0x03fc74 to 0x009171 (0x36b02 bytes) 000239 1788838477.026417 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9172): Mirror 6: 0x6a6c678e (stale) Mirror 7: 0x6a6c678e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9172): Mirror 6: 0x6a6c678e (stale) Mirror 7: 0x6a6c678e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9172: skipped 000240 1788838477.494061 write 0x0224ed thru 0x02674c (0x4260 bytes) 000241 1788838477.511551 trunc from 0x02674d to 0x02c6ef (0x5fa3 bytes) 000242 1788838477.551194 fallocate 0x032862 thru 0x03ffff (0xd79e bytes) 000243 1788838477.565639 punch from 0x01ffa4 to 0x02841c (0x8479 bytes) 000245 1788838477.597581 mapread 0x007726 thru 0x00b5ad (0x3e88 bytes) 000247 1788838477.681417 trunc from 0x02c6f0 to 0x0197fc (0x12ef3 bytes) 000248 1788838477.732354 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x197fd): Mirror 7: 0x5d32695 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x197fd): Mirror 7: 0x5d32695 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x197fd: skipped 000249 1788838478.160989 punch from 0x002cf9 to 0x0126bc (0xf9c4 bytes) 000250 1788838478.168737 write 0x012208 thru 0x01f025 (0xce1e bytes) 000251 1788838478.182247 read 0x00fd95 thru 0x01f025 (0xf291 bytes) 000252 1788838478.190610 punch from 0x007a38 to 0x011e39 (0xa402 bytes) 000253 1788838478.196573 fallocate 0x0330f3 thru 0x03ffff (0xcf0d bytes) 000255 1788838478.203391 write 0x007874 thru 0x00ef75 (0x7702 bytes) 000256 1788838478.216277 mapwrite 0x006949 thru 0x00b470 (0x4b28 bytes) 000257 1788838478.249037 mapread 0x01fa0d thru 0x0267a0 (0x6d94 bytes) 000258 1788838478.263405 trunc from 0x004225 to 0x0087df (0x45bb bytes) 000259 1788838478.269426 mapwrite 0x000970 thru 0x0063fa (0x5a8b bytes) 000260 1788838478.296339 write 0x020b65 thru 0x0295ed (0x8a89 bytes) 000261 1788838478.297763 write 0x02060d thru 0x03042c (0xfe20 bytes) 000262 1788838478.305274 fallocate 0x01eeef thru 0x020f05 (0x2017 bytes) 000263 1788838478.311686 punch from 0x028a30 to 0x030f2c (0x84fd bytes) 000265 1788838478.336933 mapwrite 0x03106a thru 0x03f482 (0xe419 bytes) 000266 1788838478.385908 read 0x0241d9 thru 0x02445f (0x287 bytes) 000267 1788838478.386877 punch from 0x03c4ef to 0x03ffff (0x3b11 bytes) 000268 1788838478.392921 fallocate 0x03eba8 thru 0x03ffff (0x1458 bytes) 000269 1788838478.398091 trunc from 0x040000 to 0x02e43b (0x11bc4 bytes) 000270 1788838478.417231 write 0x027615 thru 0x02f809 (0x81f5 bytes) 000271 1788838478.420131 write 0x00e882 thru 0x019854 (0xafd3 bytes) 000272 1788838478.423334 mapwrite 0x019098 thru 0x02471a (0xb683 bytes) 000273 1788838478.450191 trunc from 0x02f80a to 0x01ea3b (0x10dce bytes) 000274 1788838478.477180 fallocate 0x0107ec thru 0x01245b (0x1c70 bytes) 000276 1788838478.487343 mapwrite 0x01c890 thru 0x01dce1 (0x1452 bytes) 000277 1788838478.509455 read 0x01c151 thru 0x01ea3b (0x28eb bytes) 000278 1788838478.511008 read 0x0131d5 thru 0x01dde7 (0xac13 bytes) 000279 1788838478.511749 read 0x009971 thru 0x01529d (0xb92d bytes) 000280 1788838478.516950 mapread 0x0119f7 thru 0x0174eb (0x5af5 bytes) 000281 1788838478.518355 mapwrite 0x011eb5 thru 0x01b36e (0x94ba bytes) 000283 1788838478.557875 mapread 0x010df4 thru 0x017b03 (0x6d10 bytes) 000284 1788838478.562729 trunc from 0x01ea3c to 0x018d83 (0x5cb8 bytes) 000285 1788838478.607269 punch from 0x012ddc to 0x018d83 (0x5fa8 bytes) 000286 1788838478.622622 trunc from 0x018d84 to 0x0357ae (0x1ca2b bytes) 000287 1788838478.668535 MIRROR_RESYNC 2 mirrors 000288 1788838479.756909 trunc from 0x0357af to 0x0215a2 (0x1420c bytes) 000289 1788838479.822333 mapread 0x0071b9 thru 0x00c6f8 (0x5540 bytes) 000290 1788838479.865956 mapwrite 0x00958d thru 0x00d010 (0x3a84 bytes) 000291 1788838479.896465 mapwrite 0x00a79f thru 0x00d094 (0x28f6 bytes) 000292 1788838479.936220 MIRROR_RESYNC 2 mirrors 000293 1788838481.112461 mapread 0x01f7f3 thru 0x0215a2 (0x1db0 bytes) 000294 1788838481.174904 mapwrite 0x024425 thru 0x03066a (0xc246 bytes) 000296 1788838481.322193 write 0x017e1d thru 0x01fd12 (0x7ef6 bytes) 000297 1788838481.331979 trunc from 0x0034de to 0x012f5b (0xfa7e bytes) 000298 1788838481.340547 punch from 0x020bb1 to 0x02798d (0x6ddd bytes) 000299 1788838481.346349 trunc from 0x023e7e to 0x03066a (0xc7ed bytes) 000300 1788838481.358064 trunc from 0x03066b to 0x03c453 (0xbde9 bytes) 000301 1788838481.396508 trunc from 0x038d71 to 0x03c453 (0x36e3 bytes) 000302 1788838481.402787 fallocate 0x03e890 thru 0x03f47b (0xbec bytes) 000304 1788838481.409606 punch from 0x025886 to 0x031c3e (0xc3b9 bytes) 000305 1788838481.415125 read 0x01c92b thru 0x021f75 (0x564b bytes) 000306 1788838481.423529 trunc from 0x03f47c to 0x01fed6 (0x1f5a5 bytes) 000307 1788838481.447603 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1fed7): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fed7): Mirror 7: 0x4fc90f12 (stale) CRC-32 checksum value for chunk [0, 0x1fed7): Mirror 1: 0x6fea305b Mirror 8: 0x6fea305b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fed7: skipped 000308 1788838482.369974 trunc from 0x01fed7 to 0x00fcc8 (0x1020e bytes) 000309 1788838482.491407 punch from 0x000b58 to 0x00da24 (0xcecd bytes) 000311 1788838482.512556 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xfcc9): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfcc9): Mirror 7: 0xac9e7d32 (stale) Mirror 8: 0x4aa7f62c (stale) CRC-32 checksum value for chunk [0, 0xfcc9): Mirror 1: 0xe8959fa0 Mirror 9: 0xe8959fa0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfcc9: skipped 000312 1788838483.734680 punch from 0x00ce4b to 0x00fcc8 (0x2e7e bytes) 000313 1788838483.848126 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xfcc9): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfcc9): Mirror 7: 0xac9e7d32 (stale) Mirror 8: 0x4aa7f62c (stale) Mirror 9: 0xe8959fa0 (stale) CRC-32 checksum value for chunk [0, 0xfcc9): Mirror 1: 0xe8959fa0 Mirror 10: 0xe8959fa0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfcc9: skipped 000314 1788838484.942337 read 0x006272 thru 0x00fcc8 (0x9a57 bytes) 000315 1788838484.968585 fallocate 0x026163 thru 0x02838e (0x222c bytes) 000317 1788838485.041151 punch from 0x00609d to 0x00e646 (0x85aa bytes) 000318 1788838485.052785 trunc from 0x02838f to 0x000897 (0x27af7 bytes) 000319 1788838485.103229 mapwrite 0x0107e2 thru 0x018f25 (0x8744 bytes) 000320 1788838485.186029 write 0x01835e thru 0x027ddf (0xfa82 bytes) 000321 1788838485.190474 trunc from 0x027de0 to 0x0347f7 (0xca18 bytes) 000322 1788838485.220561 read 0x0243ea thru 0x02f3af (0xafc6 bytes) 000323 1788838485.249382 trunc from 0x02890d to 0x02a9ff (0x20f3 bytes) 000324 1788838485.268633 read 0x033ead thru 0x0347f7 (0x94b bytes) 000325 1788838485.271735 mapwrite 0x016f76 thru 0x01f55c (0x85e7 bytes) 000326 1788838485.309576 mapwrite 0x03dd07 thru 0x03ffff (0x22f9 bytes) 000327 1788838485.366247 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xdf7ed52d (stale) Mirror 8: 0xceda846a (stale) Mirror 9: 0x75053cf0 (stale) Mirror 10: 0x75053cf0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc3b8faf0 Mirror 11: 0xc3b8faf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000329 1788838486.482675 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xdf7ed52d (stale) Mirror 8: 0xceda846a (stale) Mirror 10: 0x75053cf0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xdf7ed52d (stale) Mirror 8: 0xceda846a (stale) Mirror 10: 0x75053cf0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc3b8faf0 Mirror 11: 0xc3b8faf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000330 1788838487.301020 mapread 0x03482c thru 0x03a152 (0x5927 bytes) 000332 1788838487.364596 trunc from 0x02b5b3 to 0x036106 (0xab54 bytes) 000333 1788838487.443337 mapread 0x03316f thru 0x03678b (0x361d bytes) 000334 1788838487.468933 trunc from 0x02b2bb to 0x035e64 (0xabaa bytes) 000335 1788838487.481917 fallocate 0x00c3e3 thru 0x00fa01 (0x361f bytes) 000337 1788838487.492346 mapread 0x02637f thru 0x029e4d (0x3acf bytes) 000338 1788838487.529330 mapread 0x004c60 thru 0x00c9a0 (0x7d41 bytes) 000339 1788838487.585139 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xdf7ed52d (stale) Mirror 8: 0xceda846a (stale) Mirror 10: 0x75053cf0 (stale) Mirror 11: 0xc3b8faf0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc3b8faf0 Mirror 12: 0xc3b8faf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000340 1788838488.888036 read 0x00b2aa thru 0x010e2b (0x5b82 bytes) 000341 1788838488.939568 punch from 0x0117f9 to 0x01ca9e (0xb2a6 bytes) 000342 1788838488.986062 fallocate 0x01f69b thru 0x02f31e (0xfc84 bytes) 000343 1788838488.991095 trunc from 0x019be4 to 0x029b13 (0xff30 bytes) 000344 1788838488.998613 trunc from 0x01a58e to 0x02188e (0x7301 bytes) 000345 1788838489.007079 write 0x002a5e thru 0x00faef (0xd092 bytes) 000346 1788838489.031060 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xceda846a (stale) Mirror 10: 0x75053cf0 (stale) Mirror 11: 0xc3b8faf0 (stale) Mirror 12: 0xc3b8faf0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xceda846a (stale) Mirror 10: 0x75053cf0 (stale) Mirror 11: 0xc3b8faf0 (stale) Mirror 12: 0xc3b8faf0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000347 1788838489.623735 read 0x01f72a thru 0x02607b (0x6952 bytes) 000348 1788838489.633133 write 0x011111 thru 0x01850f (0x73ff bytes) 000349 1788838489.688243 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xceda846a (stale) Mirror 10: 0x75053cf0 (stale) Mirror 11: 0xc3b8faf0 (stale) Mirror 12: 0xc3b8faf0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7920b138 Mirror 13: 0x7920b138 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000350 1788838490.989028 mapread 0x009164 thru 0x00d2e6 (0x4183 bytes) 000351 1788838491.044321 trunc from 0x00dfa8 to 0x014b28 (0x6b81 bytes) 000352 1788838491.156057 trunc from 0x035cf7 to 0x039199 (0x34a3 bytes) 000353 1788838491.160617 punch from 0x01b68b to 0x0273a8 (0xbd1e bytes) 000354 1788838491.175137 punch from 0x031fdf to 0x03ffff (0xe021 bytes) 000355 1788838491.185987 read 0x020a4c thru 0x0229a8 (0x1f5d bytes) 000356 1788838491.196982 mapread 0x01e2ec thru 0x02377b (0x5490 bytes) 000357 1788838491.207230 punch from 0x01fb9d to 0x029826 (0x9c8a bytes) 000358 1788838491.214899 read 0x02ff9e thru 0x038fe2 (0x9045 bytes) 000359 1788838491.251542 mapread 0x00011c thru 0x00f6ec (0xf5d1 bytes) 000360 1788838491.259982 MIRROR_RESYNC 6 mirrors 000361 1788838492.987537 mapwrite 0x00f57a thru 0x01177d (0x2204 bytes) 000362 1788838493.077229 trunc from 0x015d6a to 0x017b87 (0x1e1e bytes) 000363 1788838493.082135 read 0x007a8f thru 0x00ff4c (0x84be bytes) 000364 1788838493.090093 mapread 0x010d79 thru 0x0129bf (0x1c47 bytes) 000365 1788838493.091509 mapwrite 0x00e67a thru 0x00eb07 (0x48e bytes) 000366 1788838493.104605 fallocate 0x002fe9 thru 0x0036f1 (0x709 bytes) 000367 1788838493.109571 fallocate 0x00bc0e thru 0x00eb9e (0x2f91 bytes) 000368 1788838493.116397 trunc from 0x040000 to 0x0280ed (0x17f12 bytes) 000371 1788838493.138934 read 0x027371 thru 0x0280ed (0xd7d bytes) 000372 1788838493.167736 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x280ee): Mirror 8: 0x1212c375 (stale) Mirror 10: 0x1212c375 (stale) Mirror 11: 0x1212c375 (stale) Mirror 12: 0x1212c375 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x280ee): Mirror 8: 0x1212c375 (stale) Mirror 10: 0x1212c375 (stale) Mirror 11: 0x1212c375 (stale) Mirror 12: 0x1212c375 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x280ee: skipped 000373 1788838493.680534 mapread 0x01c0dc thru 0x0280ed (0xc012 bytes) 000374 1788838493.733351 write 0x029ad3 thru 0x036aae (0xcfdc bytes) 000375 1788838493.746516 trunc from 0x036aaf to 0x02b0eb (0xb9c3 bytes) 000376 1788838493.777471 punch from 0x01a026 to 0x023dbf (0x9d9a bytes) 000377 1788838493.790354 read 0x016114 thru 0x01f1c3 (0x90b0 bytes) 000378 1788838493.809024 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2b0ec): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b0ec): Mirror 8: 0x16008e2b (stale) Mirror 10: 0x16008e2b (stale) Mirror 11: 0x16008e2b (stale) Mirror 12: 0x16008e2b (stale) CRC-32 checksum value for chunk [0, 0x2b0ec): Mirror 1: 0x95a885b1 Mirror 13: 0x95a885b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b0ec: skipped 000379 1788838494.928018 MIRROR_RESYNC 6 mirrors 000380 1788838496.890193 punch from 0x023541 to 0x02b0eb (0x7bab bytes) 000381 1788838496.942906 punch from 0x0037c4 to 0x00baa5 (0x82e2 bytes) 000382 1788838496.951426 fallocate 0x03fc19 thru 0x03ffff (0x3e7 bytes) 000383 1788838496.962575 trunc from 0x02b0ec to 0x0188cb (0x12820 bytes) 000384 1788838497.016211 trunc from 0x0188cc to 0x016cde (0x1bed bytes) 000385 1788838497.042584 punch from 0x01010b to 0x0148a0 (0x4796 bytes) 000386 1788838497.048553 mapwrite 0x028950 thru 0x0296b0 (0xd61 bytes) 000387 1788838497.094900 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x296b1): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x296b1): Mirror 8: 0xe6a01556 (stale) Mirror 10: 0xe6a01556 (stale) Mirror 11: 0xe6a01556 (stale) Mirror 12: 0xe6a01556 (stale) Mirror 13: 0xe6a01556 (stale) CRC-32 checksum value for chunk [0, 0x296b1): Mirror 1: 0x96de14e0 Mirror 14: 0x96de14e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x296b1: skipped 000388 1788838498.420603 write 0x002a58 thru 0x00825a (0x5803 bytes) 000390 1788838498.486931 mapwrite 0x01d6c6 thru 0x0242f5 (0x6c30 bytes) 000391 1788838498.526333 read 0x00ff2a thru 0x0106f2 (0x7c9 bytes) 000392 1788838498.531970 read 0x00017b thru 0x00bbce (0xba54 bytes) 000393 1788838498.576050 MIRROR_RESYNC 7 mirrors 000394 1788838502.515298 write 0x013923 thru 0x01de48 (0xa526 bytes) 000395 1788838502.643427 punch from 0x02407c to 0x0295a3 (0x5528 bytes) 000396 1788838502.662666 mapread 0x01b581 thru 0x023b7b (0x85fb bytes) 000397 1788838502.705595 fallocate 0x02c3e9 thru 0x0395c1 (0xd1d9 bytes) 000398 1788838502.716015 trunc from 0x031e26 to 0x0395c1 (0x779c bytes) 000399 1788838502.727921 trunc from 0x02e35b to 0x038ccb (0xa971 bytes) 000400 1788838502.741669 mapread 0x033ed7 thru 0x039088 (0x51b2 bytes) 000401 1788838502.790983 write 0x0128c0 thru 0x019658 (0x6d99 bytes) 000402 1788838502.813657 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x395c2): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395c2): Mirror 8: 0xa748b116 (stale) Mirror 10: 0xa748b116 (stale) Mirror 11: 0xa748b116 (stale) Mirror 12: 0xa748b116 (stale) Mirror 13: 0xa748b116 (stale) Mirror 14: 0xa748b116 (stale) CRC-32 checksum value for chunk [0, 0x395c2): Mirror 1: 0x295500af Mirror 15: 0x295500af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395c2: skipped 000403 1788838504.163303 mapread 0x0210dc thru 0x02428b (0x31b0 bytes) 000404 1788838504.199651 trunc from 0x0395c2 to 0x017a8a (0x21b37 bytes) 000405 1788838504.255311 mapwrite 0x02760b thru 0x0292ff (0x1cf5 bytes) 000406 1788838504.325535 mapread 0x017b24 thru 0x018037 (0x514 bytes) 000407 1788838504.346188 MIRROR_RESYNC 8 mirrors 000408 1788838506.783251 trunc from 0x029300 to 0x036a4e (0xd74f bytes) 000409 1788838506.836128 trunc from 0x004a52 to 0x00f253 (0xa802 bytes) 000411 1788838506.844425 write 0x012996 thru 0x0140d7 (0x1742 bytes) 000412 1788838506.858626 MIRROR_RESYNC 8 mirrors 000413 1788838510.179303 mapread 0x01f367 thru 0x02ab09 (0xb7a3 bytes) 000414 1788838510.212556 write 0x031003 thru 0x03bbe5 (0xabe3 bytes) 000415 1788838510.251817 read 0x01745d thru 0x0206fa (0x929e bytes) 000416 1788838510.261457 write 0x03999e thru 0x03dcd7 (0x433a bytes) 000417 1788838510.263011 mapread 0x02d4e8 thru 0x02d96f (0x488 bytes) 000418 1788838510.274375 punch from 0x0340fe to 0x037dad (0x3cb0 bytes) 000419 1788838510.295291 write 0x0001f8 thru 0x004871 (0x467a bytes) 000420 1788838510.310974 write 0x01a471 thru 0x029e06 (0xf996 bytes) 000421 1788838510.319269 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dcd8): Mirror 8: 0x2a76b12f (stale) Mirror 10: 0x2a76b12f (stale) Mirror 11: 0x2a76b12f (stale) Mirror 13: 0x2a76b12f (stale) Mirror 14: 0x2a76b12f (stale) Mirror 15: 0x2a76b12f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dcd8): Mirror 8: 0x2a76b12f (stale) Mirror 10: 0x2a76b12f (stale) Mirror 11: 0x2a76b12f (stale) Mirror 13: 0x2a76b12f (stale) Mirror 14: 0x2a76b12f (stale) Mirror 15: 0x2a76b12f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dcd8: skipped 000423 1788838511.227097 MIRROR_RESYNC 7 mirrors 000424 1788838514.651486 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000425 1788838515.150570 read 0x02ff3a thru 0x0384d5 (0x859c bytes) 000426 1788838515.169998 trunc from 0x03dcd8 to 0x02bd57 (0x11f80 bytes) 000428 1788838515.264807 fallocate 0x03c940 thru 0x03ffff (0x36c0 bytes) 000429 1788838515.277593 punch from 0x02e103 to 0x02e1cc (0xca bytes) 000430 1788838515.283390 mapwrite 0x004891 thru 0x00a9a6 (0x6116 bytes) 000431 1788838515.326157 write 0x002485 thru 0x00fea7 (0xda23 bytes) 000432 1788838515.327880 write 0x00349e thru 0x00cf49 (0x9aac bytes) 000433 1788838515.330612 mapwrite 0x037012 thru 0x03bd5b (0x4d4a bytes) 000434 1788838515.362530 punch from 0x02da05 to 0x031471 (0x3a6d bytes) 000435 1788838515.368067 trunc from 0x02e3d8 to 0x032993 (0x45bc bytes) 000436 1788838515.373515 write 0x026475 thru 0x02c658 (0x61e4 bytes) 000437 1788838515.384294 write 0x003768 thru 0x005faf (0x2848 bytes) 000438 1788838515.387083 trunc from 0x040000 to 0x038a9d (0x7562 bytes) 000439 1788838515.414881 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38a9e): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a9e): Mirror 8: 0x29b033f2 (stale) Mirror 11: 0x29b033f2 (stale) Mirror 13: 0x29b033f2 (stale) Mirror 14: 0x29b033f2 (stale) Mirror 15: 0x29b033f2 (stale) CRC-32 checksum value for chunk [0, 0x38a9e): Mirror 1: 0x4d9fb346 Mirror 16: 0x4d9fb346 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a9e: skipped 000440 1788838516.966674 mapwrite 0x029b09 thru 0x031424 (0x791c bytes) 000441 1788838517.147559 trunc from 0x038a9e to 0x017d59 (0x20d44 bytes) 000442 1788838517.184199 mapwrite 0x01691a thru 0x02514e (0xe835 bytes) 000443 1788838517.252189 trunc from 0x02514f to 0x03221b (0xd0cd bytes) 000444 1788838517.272082 trunc from 0x03221c to 0x02e4f1 (0x3d2a bytes) 000445 1788838517.289838 write 0x02cc19 thru 0x0388e5 (0xbccd bytes) 000446 1788838517.301391 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x388e6): Mirror 8: 0x11cb13d5 (stale) Mirror 11: 0x11cb13d5 (stale) Mirror 14: 0x11cb13d5 (stale) Mirror 15: 0x11cb13d5 (stale) Mirror 16: 0xc878abce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x388e6): Mirror 8: 0x11cb13d5 (stale) Mirror 11: 0x11cb13d5 (stale) Mirror 14: 0x11cb13d5 (stale) Mirror 15: 0x11cb13d5 (stale) Mirror 16: 0xc878abce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x388e6: skipped 000447 1788838517.767007 read 0x01d160 thru 0x02784d (0xa6ee bytes) 000448 1788838517.780322 mapread 0x01cbcf thru 0x01d65e (0xa90 bytes) 000450 1788838517.797536 MIRROR_RESYNC 6 mirrors 000451 1788838519.484260 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000452 1788838520.086818 mapread 0x011165 thru 0x0187fe (0x769a bytes) 000453 1788838520.142576 write 0x00f439 thru 0x01a55c (0xb124 bytes) 000455 1788838520.218173 trunc from 0x01e3f1 to 0x02daa4 (0xf6b4 bytes) 000456 1788838520.228378 write 0x0057e3 thru 0x00bd5d (0x657b bytes) 000457 1788838520.253683 mapread 0x030a69 thru 0x0388e5 (0x7e7d bytes) 000458 1788838520.273177 trunc from 0x023843 to 0x0298f9 (0x60b7 bytes) 000459 1788838520.290970 fallocate 0x023155 thru 0x028894 (0x5740 bytes) 000460 1788838520.305195 trunc from 0x007527 to 0x00b057 (0x3b31 bytes) 000462 1788838520.350163 trunc from 0x027b2c to 0x029511 (0x19e6 bytes) 000463 1788838520.361078 write 0x025258 thru 0x02850b (0x32b4 bytes) 000464 1788838520.403332 read 0x015261 thru 0x01d07a (0x7e1a bytes) 000465 1788838520.409420 read 0x00af64 thru 0x013d43 (0x8de0 bytes) 000466 1788838520.420524 trunc from 0x027cc7 to 0x03190d (0x9c47 bytes) 000467 1788838520.437578 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x388e6): Mirror 8: 0x3905df0a (stale) Mirror 11: 0x3905df0a (stale) Mirror 16: 0x3905df0a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x388e6): Mirror 8: 0x3905df0a (stale) Mirror 11: 0x3905df0a (stale) Mirror 16: 0x3905df0a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x388e6: skipped 000468 1788838520.932281 punch from 0x008f0c to 0x01528f (0xc384 bytes) 000469 1788838520.946490 trunc from 0x0388e6 to 0x031028 (0x78bd bytes) 000470 1788838521.005279 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31029): Mirror 8: 0xb59fcefb (stale) Mirror 11: 0xb59fcefb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31029): Mirror 8: 0xb59fcefb (stale) Mirror 11: 0xb59fcefb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31029: skipped 000471 1788838521.565239 read 0x016047 thru 0x01a09b (0x4055 bytes) 000472 1788838521.576992 mapwrite 0x0171e4 thru 0x01c5fb (0x5418 bytes) 000473 1788838521.617428 write 0x00cd61 thru 0x012c2d (0x5ecd bytes) 000476 1788838521.657188 write 0x00353c thru 0x00e79a (0xb25f bytes) 000477 1788838521.683217 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31029): Mirror 11: 0xb59fcefb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31029): Mirror 11: 0xb59fcefb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31029: skipped 000478 1788838522.068443 trunc from 0x02a059 to 0x031028 (0x6fd0 bytes) 000479 1788838522.075614 write 0x01e806 thru 0x02a89e (0xc099 bytes) 000480 1788838522.092304 read 0x01ecb7 thru 0x02294f (0x3c99 bytes) 000481 1788838522.093037 fallocate 0x036517 thru 0x03d073 (0x6b5d bytes) 000482 1788838522.098062 read 0x024936 thru 0x02a073 (0x573e bytes) 000483 1788838522.100033 punch from 0x015808 to 0x01622e (0xa27 bytes) 000484 1788838522.104431 fallocate 0x00d8ba thru 0x010281 (0x29c8 bytes) 000485 1788838522.111929 mapread 0x008c7a thru 0x014db0 (0xc137 bytes) 000486 1788838522.125074 trunc from 0x025396 to 0x030625 (0xb290 bytes) 000487 1788838522.140973 mapread 0x01aee8 thru 0x02a5d1 (0xf6ea bytes) 000488 1788838522.167258 trunc from 0x00864f to 0x015307 (0xccb9 bytes) 000489 1788838522.173409 read 0x02abd0 thru 0x02dc06 (0x3037 bytes) 000490 1788838522.186962 write 0x015ebf thru 0x01ee86 (0x8fc8 bytes) 000492 1788838522.188946 read 0x01d244 thru 0x022296 (0x5053 bytes) 000493 1788838522.191008 read 0x014dd1 thru 0x021062 (0xc292 bytes) 000495 1788838522.196910 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000496 1788838522.385501 mapwrite 0x01e45f thru 0x02568b (0x722d bytes) 000497 1788838522.447563 fallocate 0x0005de thru 0x006459 (0x5e7c bytes) 000498 1788838522.455382 read 0x028785 thru 0x031028 (0x88a4 bytes) 000501 1788838522.480130 trunc from 0x010c26 to 0x013ccc (0x30a7 bytes) 000502 1788838522.488540 mapwrite 0x00cbe5 thru 0x018a34 (0xbe50 bytes) 000503 1788838522.530502 trunc from 0x029b71 to 0x02c2bd (0x274d bytes) 000504 1788838522.538043 read 0x027c1d thru 0x029da9 (0x218d bytes) 000505 1788838522.548069 read 0x0305e3 thru 0x031028 (0xa46 bytes) 000506 1788838522.548996 read 0x00cecb thru 0x010c17 (0x3d4d bytes) 000508 1788838522.555428 fallocate 0x02cb35 thru 0x039be6 (0xd0b2 bytes) 000509 1788838522.561940 fallocate 0x034630 thru 0x0388d6 (0x42a7 bytes) 000510 1788838522.568452 mapwrite 0x022906 thru 0x025504 (0x2bff bytes) 000511 1788838522.588730 trunc from 0x039be7 to 0x01f4fd (0x1a6e9 bytes) 000513 1788838522.614131 punch from 0x012aef to 0x01df3a (0xb44c bytes) 000514 1788838522.620321 write 0x004f04 thru 0x005cf4 (0xdf1 bytes) 000515 1788838522.621527 trunc from 0x003266 to 0x003ea1 (0xc3c bytes) 000516 1788838522.628092 fallocate 0x0012dd thru 0x010899 (0xf5bd bytes) 000517 1788838522.636422 read 0x0043d4 thru 0x0064b1 (0x20de bytes) 000518 1788838522.638322 mapwrite 0x00b04f thru 0x01062e (0x55e0 bytes) 000519 1788838522.655873 mapread 0x013751 thru 0x017674 (0x3f24 bytes) 000521 1788838522.666290 trunc from 0x0126a0 to 0x01eda0 (0xc701 bytes) 000523 1788838522.673399 write 0x00273d thru 0x00a464 (0x7d28 bytes) 000524 1788838522.675203 mapwrite 0x03849e thru 0x03ffff (0x7b62 bytes) 000525 1788838522.720199 punch from 0x029ef0 to 0x031a9a (0x7bab bytes) 000526 1788838522.725874 trunc from 0x007f2a to 0x00b6a9 (0x3780 bytes) 000527 1788838522.732167 mapread 0x0255df thru 0x02cf60 (0x7982 bytes) 000528 1788838522.751500 trunc from 0x040000 to 0x03e2e3 (0x1d1c bytes) 000529 1788838522.768094 write 0x0363f7 thru 0x03b9bc (0x55c6 bytes) 000531 1788838522.775357 MIRROR_EXTEND to 2 mirrors 000532 1788838523.754924 mapwrite 0x00375f thru 0x00705b (0x38fd bytes) 000533 1788838523.883248 read 0x009b68 thru 0x00f247 (0x56e0 bytes) 000534 1788838523.885083 trunc from 0x03e2e4 to 0x025db8 (0x1852b bytes) 000535 1788838523.951358 trunc from 0x025db9 to 0x00ff1e (0x15e9a bytes) 000536 1788838523.987248 trunc from 0x00fbc7 to 0x00ff1e (0x358 bytes) 000537 1788838523.996619 read 0x000512 thru 0x00215f (0x1c4e bytes) 000538 1788838524.007977 mapread 0x00fa15 thru 0x00ff1e (0x50a bytes) 000539 1788838524.012663 read 0x00726f thru 0x00dd03 (0x6a95 bytes) 000540 1788838524.020181 mapread 0x00c8db thru 0x00ff1e (0x3644 bytes) 000541 1788838524.021143 mapwrite 0x01034a thru 0x0181d2 (0x7e89 bytes) 000542 1788838524.147896 trunc from 0x0181d3 to 0x012c70 (0x5562 bytes) 000543 1788838524.189123 write 0x01a997 thru 0x024815 (0x9e7f bytes) 000544 1788838524.190895 trunc from 0x022163 to 0x024815 (0x26b3 bytes) 000545 1788838524.207518 mapread 0x014726 thru 0x01dd07 (0x95e2 bytes) 000546 1788838524.270273 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24816): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24816): Mirror 2: 0xc6f8edf (stale) CRC-32 checksum value for chunk [0, 0x24816): Mirror 1: 0xabcefc71 Mirror 3: 0xabcefc71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24816: skipped 000547 1788838525.710558 MIRROR_RESYNC 3 mirrors 000548 1788838526.663832 read 0x011ce0 thru 0x017bf5 (0x5f16 bytes) 000550 1788838526.687479 fallocate 0x01789b thru 0x022714 (0xae7a bytes) 000551 1788838526.795647 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24816): Mirror 2: 0xabcefc71 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24816): Mirror 2: 0xabcefc71 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24816: skipped 000552 1788838527.389997 read 0x010cf7 thru 0x0187a1 (0x7aab bytes) 000553 1788838527.418253 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24816): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24816): Mirror 2: 0xabcefc71 (stale) CRC-32 checksum value for chunk [0, 0x24816): Mirror 1: 0xabcefc71 Mirror 3: 0xabcefc71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24816: skipped 000554 1788838528.806159 write 0x03165b thru 0x033210 (0x1bb6 bytes) 000555 1788838528.868785 trunc from 0x00baba to 0x00bfd9 (0x520 bytes) 000556 1788838528.875823 punch from 0x0109ee to 0x013101 (0x2714 bytes) 000558 1788838528.885859 punch from 0x01c652 to 0x020941 (0x42f0 bytes) 000559 1788838528.895533 read 0x0163ed thru 0x01c3e4 (0x5ff8 bytes) 000560 1788838528.923636 MIRROR_RESYNC 3 mirrors 000562 1788838530.509669 MIRROR_RESYNC 3 mirrors 000563 1788838531.222216 read 0x002607 thru 0x007738 (0x5132 bytes) 000564 1788838531.238076 trunc from 0x033211 to 0x002848 (0x309c8 bytes) 000565 1788838531.293314 trunc from 0x002705 to 0x002848 (0x144 bytes) 000566 1788838531.299535 trunc from 0x00213d to 0x002848 (0x70c bytes) 000567 1788838531.309357 trunc from 0x001412 to 0x002848 (0x1437 bytes) 000568 1788838531.317408 read 0x00115e thru 0x002848 (0x16eb bytes) 000569 1788838531.326917 write 0x02d979 thru 0x039460 (0xbae8 bytes) 000571 1788838531.334297 read 0x019996 thru 0x021dd0 (0x843b bytes) 000572 1788838531.349987 mapread 0x01c6da thru 0x020f1b (0x4842 bytes) 000573 1788838531.351297 trunc from 0x039461 to 0x03f176 (0x5d16 bytes) 000574 1788838531.378072 punch from 0x00c904 to 0x00f1ed (0x28ea bytes) 000575 1788838531.383564 trunc from 0x03f177 to 0x0259d7 (0x1979f bytes) 000576 1788838531.407824 mapwrite 0x02667d thru 0x034a22 (0xe3a6 bytes) 000577 1788838531.580611 MIRROR_RESYNC 3 mirrors 000578 1788838532.407863 mapwrite 0x01f204 thru 0x0270ec (0x7ee9 bytes) 000579 1788838532.543728 trunc from 0x034a23 to 0x022095 (0x1298d bytes) 000580 1788838532.584263 write 0x030de7 thru 0x032c53 (0x1e6d bytes) 000581 1788838532.590640 write 0x01ddbe thru 0x022832 (0x4a75 bytes) 000582 1788838532.594126 trunc from 0x02e337 to 0x0314c2 (0x318c bytes) 000583 1788838532.621524 fallocate 0x00a9cc thru 0x00e27c (0x38b1 bytes) 000584 1788838532.637408 punch from 0x006b63 to 0x012b0e (0xbfac bytes) 000585 1788838532.650157 mapwrite 0x016ff4 thru 0x017682 (0x68f bytes) 000587 1788838532.701678 fallocate 0x03205b thru 0x0388e7 (0x688d bytes) 000588 1788838532.712124 mapread 0x0128ab thru 0x021803 (0xef59 bytes) 000591 1788838532.726134 write 0x007510 thru 0x0167ca (0xf2bb bytes) 000592 1788838532.743824 punch from 0x031fa4 to 0x0388e7 (0x6944 bytes) 000593 1788838532.750534 mapwrite 0x023793 thru 0x028d19 (0x5587 bytes) 000594 1788838532.806110 trunc from 0x0388e8 to 0x00bdf0 (0x2caf7 bytes) 000595 1788838532.830275 punch from 0x0087d5 to 0x00bdf0 (0x361c bytes) 000596 1788838532.845949 mapread 0x0072e8 thru 0x00bdf0 (0x4b09 bytes) 000597 1788838532.866925 trunc from 0x00bdf1 to 0x003a6d (0x8383 bytes) 000598 1788838532.895938 read 0x0018ff thru 0x003a6d (0x216f bytes) 000599 1788838532.896797 trunc from 0x00297c to 0x003a6d (0x10f2 bytes) 000601 1788838532.905561 fallocate 0x01f6fe thru 0x0212e5 (0x1be8 bytes) 000602 1788838532.912902 mapread 0x01beee thru 0x0212e5 (0x53f8 bytes) 000603 1788838532.931678 write 0x023b00 thru 0x02acd3 (0x71d4 bytes) 000605 1788838532.935810 punch from 0x0210fe to 0x02acd3 (0x9bd6 bytes) 000606 1788838532.954898 trunc from 0x02acd4 to 0x0163af (0x14924 bytes) 000607 1788838532.970299 mapwrite 0x00e9b7 thru 0x00f094 (0x6de bytes) 000608 1788838532.992981 mapwrite 0x029774 thru 0x02a036 (0x8c3 bytes) 000609 1788838533.035606 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a037): Mirror 3: 0xa5bce22f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a037): Mirror 3: 0xa5bce22f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a037: skipped 000610 1788838533.566217 trunc from 0x02a037 to 0x031420 (0x73ea bytes) 000612 1788838533.594916 trunc from 0x031421 to 0x03fa99 (0xe679 bytes) 000613 1788838533.616741 trunc from 0x019417 to 0x0279f9 (0xe5e3 bytes) 000614 1788838533.624874 read 0x02c5b2 thru 0x033661 (0x70b0 bytes) 000615 1788838533.634569 fallocate 0x005297 thru 0x00fc3b (0xa9a5 bytes) 000616 1788838533.640101 punch from 0x02c647 to 0x02e5c8 (0x1f82 bytes) 000617 1788838533.644864 read 0x00fdc6 thru 0x01cd07 (0xcf42 bytes) 000618 1788838533.664603 read 0x00f4ee thru 0x014444 (0x4f57 bytes) 000619 1788838533.667675 write 0x011d24 thru 0x01e4e4 (0xc7c1 bytes) 000620 1788838533.672952 fallocate 0x027b94 thru 0x02a251 (0x26be bytes) 000621 1788838533.680630 punch from 0x0156b5 to 0x022f96 (0xd8e2 bytes) 000622 1788838533.712831 mapwrite 0x0242a9 thru 0x02a8b7 (0x660f bytes) 000623 1788838533.739495 punch from 0x006514 to 0x00bd3a (0x5827 bytes) 000624 1788838533.746593 read 0x022306 thru 0x02689c (0x4597 bytes) 000625 1788838533.759888 trunc from 0x03fa9a to 0x003e77 (0x3bc22 bytes) 000626 1788838533.781210 read 0x002a46 thru 0x003e77 (0x1432 bytes) 000627 1788838533.784907 mapread 0x00383e thru 0x003e77 (0x63a bytes) 000628 1788838533.786245 read 0x003c5a thru 0x003e77 (0x21e bytes) 000629 1788838533.795616 read 0x00191d thru 0x003e77 (0x255b bytes) 000630 1788838533.802430 mapwrite 0x00a5c7 thru 0x0164cf (0xbf09 bytes) 000631 1788838533.894841 write 0x01d880 thru 0x029bc0 (0xc341 bytes) 000632 1788838533.898804 write 0x01bb03 thru 0x02198a (0x5e88 bytes) 000633 1788838533.920247 read 0x00bda6 thru 0x013179 (0x73d4 bytes) 000634 1788838533.921011 fallocate 0x028df3 thru 0x035024 (0xc232 bytes) 000635 1788838533.928124 punch from 0x016c22 to 0x02271b (0xbafa bytes) 000636 1788838533.943668 punch from 0x00b48b to 0x0135fc (0x8172 bytes) 000637 1788838533.951270 mapread 0x018dff thru 0x026c93 (0xde95 bytes) 000638 1788838533.975539 trunc from 0x028d68 to 0x029bc0 (0xe59 bytes) 000639 1788838533.980472 write 0x037042 thru 0x03ffff (0x8fbe bytes) 000641 1788838533.983798 mapread 0x012954 thru 0x0227cc (0xfe79 bytes) 000642 1788838533.985448 trunc from 0x040000 to 0x0144d4 (0x2bb2b bytes) 000643 1788838534.020032 trunc from 0x0038dd to 0x012fc7 (0xf6eb bytes) 000644 1788838534.025947 fallocate 0x006913 thru 0x00f43a (0x8b28 bytes) 000645 1788838534.034887 fallocate 0x01f0c2 thru 0x02567a (0x65b9 bytes) 000646 1788838534.040627 punch from 0x00a3fb to 0x0118fa (0x7500 bytes) 000647 1788838534.048278 mapread 0x01a3e6 thru 0x02567a (0xb295 bytes) 000648 1788838534.130182 mapwrite 0x03af1c thru 0x03ffff (0x50e4 bytes) 000649 1788838534.232520 write 0x035458 thru 0x03ffff (0xaba8 bytes) 000650 1788838534.234084 read 0x00a040 thru 0x00fe9b (0x5e5c bytes) 000651 1788838534.259890 read 0x004595 thru 0x00afca (0x6a36 bytes) 000652 1788838534.260763 trunc from 0x040000 to 0x0375f7 (0x8a08 bytes) 000653 1788838534.289045 punch from 0x004744 to 0x012e6f (0xe72c bytes) 000654 1788838534.306670 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x375f8): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x375f8): Mirror 3: 0x82e6c40 (stale) CRC-32 checksum value for chunk [0, 0x375f8): Mirror 1: 0x1a18f981 Mirror 4: 0x1a18f981 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x375f8: skipped 000655 1788838535.508121 read 0x02fa12 thru 0x0375f7 (0x7be6 bytes) 000656 1788838535.553054 MIRROR_RESYNC 3 mirrors 000658 1788838536.527034 fallocate 0x032741 thru 0x0360fb (0x39bb bytes) 000659 1788838536.606944 write 0x037ea4 thru 0x03e30b (0x6468 bytes) 000660 1788838536.621575 trunc from 0x01b71e to 0x02a88b (0xf16e bytes) 000661 1788838536.640113 fallocate 0x030dfe thru 0x039d8f (0x8f92 bytes) 000662 1788838536.649996 write 0x037b2e thru 0x03ffff (0x84d2 bytes) 000663 1788838536.655393 read 0x0340bd thru 0x03f99d (0xb8e1 bytes) 000664 1788838536.668921 mapwrite 0x02667f thru 0x029a8d (0x340f bytes) 000665 1788838536.777229 trunc from 0x00f9c0 to 0x018382 (0x89c3 bytes) 000666 1788838536.787036 mapread 0x03a9eb thru 0x03ffff (0x5615 bytes) 000667 1788838536.787898 trunc from 0x040000 to 0x0137e4 (0x2c81b bytes) 000668 1788838536.916249 MIRROR_RESYNC 3 mirrors 000669 1788838537.827301 trunc from 0x0137e5 to 0x024322 (0x10b3e bytes) 000670 1788838537.915193 write 0x0366ea thru 0x03ffff (0x9916 bytes) 000671 1788838537.917223 trunc from 0x040000 to 0x011261 (0x2ed9e bytes) 000672 1788838537.949916 mapread 0x0086b1 thru 0x00e581 (0x5ed1 bytes) 000673 1788838537.985377 read 0x00e239 thru 0x011261 (0x3029 bytes) 000674 1788838537.986093 read 0x008340 thru 0x011261 (0x8f22 bytes) 000677 1788838537.992222 trunc from 0x00a248 to 0x011261 (0x701a bytes) 000678 1788838538.000796 fallocate 0x01421f thru 0x01b691 (0x7473 bytes) 000679 1788838538.006221 read 0x006bcd thru 0x011261 (0xa695 bytes) 000680 1788838538.016161 punch from 0x00fb4d to 0x011261 (0x1715 bytes) 000681 1788838538.023730 mapread 0x0098f3 thru 0x011261 (0x796f bytes) 000682 1788838538.057227 mapwrite 0x0335ac thru 0x03b59d (0x7ff2 bytes) 000683 1788838538.099543 mapread 0x006a25 thru 0x0084fe (0x1ada bytes) 000684 1788838538.101189 read 0x015649 thru 0x01da0f (0x83c7 bytes) 000685 1788838538.102900 trunc from 0x03b59e to 0x033630 (0x7f6d bytes) 000686 1788838538.120544 trunc from 0x004728 to 0x00b325 (0x6bfe bytes) 000687 1788838538.126009 punch from 0x032424 to 0x033630 (0x120d bytes) 000688 1788838538.130842 fallocate 0x00b437 thru 0x018d88 (0xd952 bytes) 000689 1788838538.136478 fallocate 0x0190b1 thru 0x01c757 (0x36a7 bytes) 000690 1788838538.141537 trunc from 0x033631 to 0x007dac (0x2b884 bytes) 000691 1788838538.160241 punch from 0x0057c3 to 0x007dac (0x25ea bytes) 000692 1788838538.166626 write 0x03724d thru 0x0380de (0xe92 bytes) 000693 1788838538.168965 mapwrite 0x00beca thru 0x01a6fb (0xe832 bytes) 000694 1788838538.200146 fallocate 0x02c1fa thru 0x02e3da (0x21e1 bytes) 000695 1788838538.206635 trunc from 0x03581e to 0x0380de (0x28c1 bytes) 000696 1788838538.214749 fallocate 0x03412f thru 0x035d14 (0x1be6 bytes) 000697 1788838538.220146 trunc from 0x0380df to 0x030ef7 (0x71e7 bytes) 000698 1788838538.235010 trunc from 0x030ef8 to 0x007dec (0x2910b bytes) 000700 1788838538.257157 mapwrite 0x0061c8 thru 0x00d429 (0x7262 bytes) 000701 1788838538.328495 mapwrite 0x0203a1 thru 0x024cd5 (0x4935 bytes) 000702 1788838538.381679 mapwrite 0x025901 thru 0x0274b9 (0x1bb9 bytes) 000704 1788838538.436323 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x274ba): Mirror 4: 0xc2e90f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x274ba): Mirror 4: 0xc2e90f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x274ba: skipped 000705 1788838538.755183 trunc from 0x0274ba to 0x03c283 (0x14dca bytes) 000706 1788838538.777637 punch from 0x034c08 to 0x03c283 (0x767c bytes) 000707 1788838538.783301 read 0x00789c thru 0x016b49 (0xf2ae bytes) 000708 1788838538.800952 punch from 0x021a44 to 0x0316fd (0xfcba bytes) 000709 1788838538.810773 punch from 0x031da8 to 0x03c283 (0xa4dc bytes) 000710 1788838538.823478 punch from 0x016a12 to 0x017569 (0xb58 bytes) 000713 1788838538.844902 punch from 0x020dcc to 0x02edf3 (0xe028 bytes) 000714 1788838538.864641 punch from 0x0317f4 to 0x039e15 (0x8622 bytes) 000716 1788838538.878141 trunc from 0x03c284 to 0x028d81 (0x13502 bytes) 000717 1788838538.916544 punch from 0x00b44b to 0x011b1f (0x66d5 bytes) 000718 1788838538.930222 punch from 0x0212b3 to 0x028d81 (0x7acf bytes) 000719 1788838538.941049 mapread 0x021479 thru 0x023a27 (0x25af bytes) 000720 1788838538.987464 read 0x00d9fc thru 0x018add (0xb0e2 bytes) 000721 1788838539.036393 mapread 0x016d68 thru 0x0183a8 (0x1641 bytes) 000722 1788838539.040446 read 0x00ea2b thru 0x01c46d (0xda43 bytes) 000723 1788838539.045440 trunc from 0x028d82 to 0x039459 (0x106d8 bytes) 000724 1788838539.066453 trunc from 0x03945a to 0x02e68e (0xadcb bytes) 000725 1788838539.087683 trunc from 0x02e296 to 0x02e68e (0x3f9 bytes) 000726 1788838539.097436 MIRROR_RESYNC 2 mirrors 000727 1788838540.118835 mapwrite 0x020279 thru 0x02766b (0x73f3 bytes) 000728 1788838540.227374 write 0x00abe5 thru 0x00bdd3 (0x11ef bytes) 000729 1788838540.239584 mapwrite 0x037913 thru 0x0399c8 (0x20b6 bytes) 000730 1788838540.341377 trunc from 0x0399c9 to 0x0170c2 (0x22906 bytes) 000731 1788838540.367899 write 0x03c29d thru 0x03ffff (0x3d63 bytes) 000732 1788838540.369267 trunc from 0x013105 to 0x0146a0 (0x159c bytes) 000733 1788838540.377565 mapwrite 0x029f96 thru 0x02b770 (0x17db bytes) 000734 1788838540.438861 MIRROR_RESYNC 2 mirrors 000735 1788838541.462975 read 0x03349f thru 0x03b1f7 (0x7d59 bytes) 000736 1788838541.476490 mapread 0x02f857 thru 0x037fe8 (0x8792 bytes) 000737 1788838541.516894 punch from 0x006d0c to 0x016716 (0xfa0b bytes) 000738 1788838541.643084 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000739 1788838541.919744 fallocate 0x0162f9 thru 0x01a56a (0x4272 bytes) 000740 1788838541.936928 mapwrite 0x00bd20 thru 0x0149a7 (0x8c88 bytes) 000741 1788838542.005138 mapwrite 0x03f809 thru 0x03ffff (0x7f7 bytes) 000742 1788838542.043655 read 0x02be71 thru 0x0372d9 (0xb469 bytes) 000744 1788838542.058917 trunc from 0x031681 to 0x033d49 (0x26c9 bytes) 000745 1788838542.064907 mapread 0x017591 thru 0x0234de (0xbf4e bytes) 000746 1788838542.091832 write 0x00b373 thru 0x015294 (0x9f22 bytes) 000747 1788838542.099549 fallocate 0x0394dc thru 0x03cbc3 (0x36e8 bytes) 000748 1788838542.106042 punch from 0x030d27 to 0x03a1cf (0x94a9 bytes) 000749 1788838542.120154 MIRROR_EXTEND to 2 mirrors 000750 1788838543.339899 mapwrite 0x0036b7 thru 0x0076ad (0x3ff7 bytes) 000751 1788838543.468502 write 0x00e68e thru 0x015e15 (0x7788 bytes) 000752 1788838543.480330 read 0x02bd2c thru 0x02cfc2 (0x1297 bytes) 000753 1788838543.489213 fallocate 0x01adf5 thru 0x025b2e (0xad3a bytes) 000754 1788838543.496999 punch from 0x007892 to 0x00cf26 (0x5695 bytes) 000755 1788838543.504644 trunc from 0x008eca to 0x01550b (0xc642 bytes) 000756 1788838543.523563 read 0x021bd3 thru 0x03084d (0xec7b bytes) 000758 1788838543.541346 MIRROR_RESYNC 2 mirrors 000759 1788838544.202281 mapread 0x003f41 thru 0x00fdb5 (0xbe75 bytes) 000760 1788838544.224827 mapwrite 0x00573e thru 0x00c61c (0x6edf bytes) 000761 1788838544.300481 mapread 0x03185b thru 0x039def (0x8595 bytes) 000762 1788838544.311791 write 0x02d0e8 thru 0x039699 (0xc5b2 bytes) 000763 1788838544.314532 read 0x01ea32 thru 0x0219ea (0x2fb9 bytes) 000764 1788838544.322892 mapwrite 0x023e9b thru 0x02dd33 (0x9e99 bytes) 000765 1788838544.362893 mapread 0x03579e thru 0x03ffff (0xa862 bytes) 000766 1788838544.377684 punch from 0x022c93 to 0x02e239 (0xb5a7 bytes) 000767 1788838544.387943 fallocate 0x009bce thru 0x00ace9 (0x111c bytes) 000768 1788838544.395581 mapread 0x0301d9 thru 0x03d9c7 (0xd7ef bytes) 000770 1788838544.398852 write 0x020d36 thru 0x022218 (0x14e3 bytes) 000771 1788838544.406635 punch from 0x023582 to 0x02b71c (0x819b bytes) 000773 1788838544.412916 write 0x01fd02 thru 0x02048e (0x78d bytes) 000774 1788838544.414721 punch from 0x028fe6 to 0x03855a (0xf575 bytes) 000775 1788838544.421301 mapread 0x0179f4 thru 0x01d08b (0x5698 bytes) 000776 1788838544.436769 MIRROR_RESYNC 2 mirrors 000778 1788838545.168505 punch from 0x007f6f to 0x00bc47 (0x3cd9 bytes) 000780 1788838545.215872 read 0x00a647 thru 0x017b68 (0xd522 bytes) 000781 1788838545.234549 trunc from 0x040000 to 0x03233b (0xdcc4 bytes) 000782 1788838545.304276 trunc from 0x03233c to 0x003d03 (0x2e638 bytes) 000783 1788838545.360502 punch from 0x0010a3 to 0x003d03 (0x2c61 bytes) 000784 1788838545.374259 mapread 0x001f76 thru 0x003d03 (0x1d8e bytes) 000785 1788838545.419569 mapwrite 0x01f9be thru 0x025181 (0x57c4 bytes) 000786 1788838545.476495 trunc from 0x025182 to 0x02bb85 (0x6a04 bytes) 000787 1788838545.498563 mapwrite 0x03efdf thru 0x03ffff (0x1021 bytes) 000788 1788838545.548940 read 0x02662d thru 0x035af3 (0xf4c7 bytes) 000790 1788838545.575851 trunc from 0x040000 to 0x02b699 (0x14966 bytes) 000792 1788838545.603016 mapwrite 0x01b9c4 thru 0x024546 (0x8b83 bytes) 000793 1788838545.659140 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000794 1788838545.892572 mapread 0x011ac2 thru 0x0154e8 (0x3a27 bytes) 000795 1788838545.925070 trunc from 0x02b69a to 0x01f18f (0xc50a bytes) 000796 1788838545.970040 punch from 0x000980 to 0x00d087 (0xc708 bytes) 000797 1788838546.000959 MIRROR_EXTEND to 2 mirrors 000798 1788838546.817044 trunc from 0x01f190 to 0x01cf9c (0x21f3 bytes) 000799 1788838546.866235 trunc from 0x0111d7 to 0x012e26 (0x1c50 bytes) 000800 1788838546.873890 mapwrite 0x035589 thru 0x03a98d (0x5405 bytes) 000801 1788838546.963169 trunc from 0x03a98e to 0x029e01 (0x10b8c bytes) 000802 1788838546.999829 mapread 0x00d234 thru 0x0111e3 (0x3fb0 bytes) 000803 1788838547.028284 read 0x01160f thru 0x014a21 (0x3413 bytes) 000804 1788838547.029331 write 0x03e78f thru 0x03ffff (0x1871 bytes) 000805 1788838547.040412 trunc from 0x0091ec to 0x00fdae (0x6bc3 bytes) 000806 1788838547.052045 punch from 0x0216cf to 0x024d91 (0x36c3 bytes) 000807 1788838547.064217 write 0x026046 thru 0x02fb69 (0x9b24 bytes) 000808 1788838547.094308 trunc from 0x003eff to 0x00888d (0x498f bytes) 000809 1788838547.110440 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000810 1788838547.271869 write 0x02763e thru 0x02c19f (0x4b62 bytes) 000811 1788838547.295117 fallocate 0x009147 thru 0x011a17 (0x88d1 bytes) 000813 1788838547.307128 fallocate 0x03ac1b thru 0x03ffff (0x53e5 bytes) 000814 1788838547.315662 mapread 0x038ac0 thru 0x03ffff (0x7540 bytes) 000815 1788838547.350490 trunc from 0x040000 to 0x021228 (0x1edd7 bytes) 000817 1788838547.378187 trunc from 0x006de8 to 0x016bf2 (0xfe0b bytes) 000818 1788838547.390468 write 0x032609 thru 0x03e2fc (0xbcf4 bytes) 000819 1788838547.392329 fallocate 0x03c367 thru 0x03efcd (0x2c67 bytes) 000820 1788838547.399349 trunc from 0x014dc0 to 0x01ca4d (0x7c8e bytes) 000822 1788838547.411584 punch from 0x02549a to 0x0313b7 (0xbf1e bytes) 000823 1788838547.423337 mapread 0x00838f thru 0x00e11a (0x5d8c bytes) 000824 1788838547.441629 write 0x023045 thru 0x02911c (0x60d8 bytes) 000825 1788838547.471045 mapread 0x0028fd thru 0x011a90 (0xf194 bytes) 000826 1788838547.496780 mapwrite 0x002e0c thru 0x00629d (0x3492 bytes) 000827 1788838547.537237 mapread 0x001c20 thru 0x011826 (0xfc07 bytes) 000828 1788838547.539267 punch from 0x017e90 to 0x01e0ad (0x621e bytes) 000829 1788838547.544194 read 0x03741f thru 0x03d73f (0x6321 bytes) 000830 1788838547.544943 trunc from 0x015f3e to 0x01ed28 (0x8deb bytes) 000831 1788838547.549552 punch from 0x036282 to 0x03b668 (0x53e7 bytes) 000832 1788838547.556359 punch from 0x01925d to 0x01bd38 (0x2adc bytes) 000833 1788838547.562028 fallocate 0x018b77 thru 0x027b4c (0xefd6 bytes) 000834 1788838547.568661 trunc from 0x033986 to 0x03a55e (0x6bd9 bytes) 000836 1788838547.576676 write 0x01946c thru 0x01d6e1 (0x4276 bytes) 000838 1788838547.588506 mapread 0x031e68 thru 0x03d0b9 (0xb252 bytes) 000840 1788838547.611041 read 0x01aa12 thru 0x020151 (0x5740 bytes) 000841 1788838547.627950 fallocate 0x037062 thru 0x03dac1 (0x6a60 bytes) 000842 1788838547.642547 MIRROR_EXTEND to 2 mirrors 000843 1788838548.606461 punch from 0x01d750 to 0x024d3c (0x75ed bytes) 000844 1788838548.648569 MIRROR_RESYNC 2 mirrors 000845 1788838549.527861 trunc from 0x0046ff to 0x00a2f8 (0x5bfa bytes) 000846 1788838549.585252 mapread 0x00510e thru 0x01329d (0xe190 bytes) 000847 1788838549.651539 mapwrite 0x036d95 thru 0x03b472 (0x46de bytes) 000848 1788838549.697681 read 0x027f81 thru 0x029216 (0x1296 bytes) 000849 1788838549.704014 punch from 0x019f8d to 0x01ad1c (0xd90 bytes) 000850 1788838549.710561 write 0x036857 thru 0x03d702 (0x6eac bytes) 000851 1788838549.712661 fallocate 0x003e5e thru 0x00b681 (0x7824 bytes) 000852 1788838549.727451 write 0x003468 thru 0x00c9bd (0x9556 bytes) 000853 1788838549.729599 trunc from 0x01e7f9 to 0x02c1e8 (0xd9f0 bytes) 000854 1788838549.740176 read 0x020892 thru 0x027cf5 (0x7464 bytes) 000855 1788838549.760645 write 0x01a6c8 thru 0x027f33 (0xd86c bytes) 000856 1788838549.763188 punch from 0x00a8ea to 0x015282 (0xa999 bytes) 000858 1788838549.792141 trunc from 0x03e2fd to 0x00d54e (0x30dae bytes) 000859 1788838549.817116 fallocate 0x02e0fc thru 0x034522 (0x6427 bytes) 000861 1788838549.825022 trunc from 0x0053c4 to 0x00d54e (0x818b bytes) 000862 1788838549.838901 MIRROR_RESYNC 2 mirrors 000863 1788838550.721987 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000864 1788838550.889341 trunc from 0x00d54f to 0x008b6f (0x49df bytes) 000865 1788838550.946864 write 0x01e875 thru 0x02d862 (0xefee bytes) 000866 1788838550.948515 trunc from 0x02d863 to 0x010b5e (0x1cd04 bytes) 000867 1788838550.993233 read 0x010af6 thru 0x010b5e (0x69 bytes) 000868 1788838550.999777 trunc from 0x010b5f to 0x00eca8 (0x1eb6 bytes) 000869 1788838551.027012 mapwrite 0x011a13 thru 0x01b9b8 (0x9fa6 bytes) 000870 1788838551.102501 write 0x03c976 thru 0x03f595 (0x2c20 bytes) 000871 1788838551.104025 trunc from 0x0070d9 to 0x00ce4c (0x5d74 bytes) 000872 1788838551.110458 trunc from 0x02e9d2 to 0x030769 (0x1d98 bytes) 000873 1788838551.117672 punch from 0x01e641 to 0x0292ce (0xac8e bytes) 000874 1788838551.128600 punch from 0x02b9d0 to 0x03aff2 (0xf623 bytes) 000875 1788838551.133909 trunc from 0x01ef75 to 0x02ef55 (0xffe1 bytes) 000876 1788838551.139612 punch from 0x02f3ac to 0x03f16a (0xfdbf bytes) 000877 1788838551.156912 read 0x00cf59 thru 0x01ba70 (0xeb18 bytes) 000878 1788838551.201629 write 0x03a236 thru 0x03d392 (0x315d bytes) 000880 1788838551.204353 mapwrite 0x03f1ce thru 0x03ffff (0xe32 bytes) 000881 1788838551.250075 read 0x004eef thru 0x014be2 (0xfcf4 bytes) 000882 1788838551.250794 fallocate 0x013e41 thru 0x018b62 (0x4d22 bytes) 000883 1788838551.263172 mapwrite 0x026137 thru 0x02c374 (0x623e bytes) 000885 1788838551.293445 trunc from 0x032f68 to 0x039694 (0x672d bytes) 000886 1788838551.304542 write 0x0234f1 thru 0x024711 (0x1221 bytes) 000888 1788838551.307214 read 0x025a41 thru 0x035a3a (0xfffa bytes) 000889 1788838551.322258 fallocate 0x0174f1 thru 0x01f65c (0x816c bytes) 000890 1788838551.341003 MIRROR_EXTEND to 2 mirrors 000891 1788838552.135368 fallocate 0x03c003 thru 0x03ffff (0x3ffd bytes) 000892 1788838552.182663 trunc from 0x02cb4b to 0x035fb6 (0x946c bytes) 000893 1788838552.188974 mapwrite 0x02eb4f thru 0x03c1f3 (0xd6a5 bytes) 000894 1788838552.258526 mapread 0x03b4a2 thru 0x03ffff (0x4b5e bytes) 000895 1788838552.260979 mapwrite 0x004787 thru 0x00c5f3 (0x7e6d bytes) 000897 1788838552.307008 trunc from 0x031d78 to 0x03dead (0xc136 bytes) 000898 1788838552.317429 trunc from 0x040000 to 0x024c85 (0x1b37a bytes) 000899 1788838552.378358 trunc from 0x00a2e8 to 0x018334 (0xe04d bytes) 000900 1788838552.385928 mapwrite 0x00fcda thru 0x01e068 (0xe38f bytes) 000901 1788838552.444920 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24c86): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c86): Mirror 2: 0x7c730d15 (stale) CRC-32 checksum value for chunk [0, 0x24c86): Mirror 1: 0x2776ac60 Mirror 3: 0x2776ac60 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c86: skipped 000902 1788838553.506376 trunc from 0x024c86 to 0x03e024 (0x1939f bytes) 000903 1788838553.672040 trunc from 0x03e025 to 0x009c68 (0x343bc bytes) 000904 1788838553.708997 punch from 0x009a48 to 0x009c68 (0x221 bytes) 000905 1788838553.720363 write 0x010b74 thru 0x016ea0 (0x632d bytes) 000906 1788838553.722360 read 0x014340 thru 0x016ea0 (0x2b61 bytes) 000907 1788838553.723220 trunc from 0x00f546 to 0x016ea0 (0x795b bytes) 000908 1788838553.762462 punch from 0x0031d7 to 0x005e68 (0x2c92 bytes) 000909 1788838553.770494 trunc from 0x006096 to 0x00b1d0 (0x513b bytes) 000910 1788838553.779799 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16ea1): Mirror 3: 0x5ed77385 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16ea1): Mirror 3: 0x5ed77385 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16ea1: skipped 000911 1788838554.253322 mapread 0x0099f4 thru 0x00bd9d (0x23aa bytes) 000912 1788838554.285284 read 0x00a454 thru 0x00fae5 (0x5692 bytes) 000913 1788838554.288484 read 0x00734c thru 0x013eed (0xcba2 bytes) 000914 1788838554.296685 write 0x0344af thru 0x03fbf9 (0xb74b bytes) 000915 1788838554.308430 fallocate 0x00d1aa thru 0x00f68b (0x24e2 bytes) 000916 1788838554.319605 trunc from 0x0354f4 to 0x03fbf9 (0xa706 bytes) 000918 1788838554.373593 MIRROR_RESYNC 2 mirrors 000919 1788838555.534617 mapread 0x02299b thru 0x02f6b8 (0xcd1e bytes) 000921 1788838555.567561 mapwrite 0x03e067 thru 0x03ffff (0x1f99 bytes) 000923 1788838555.623584 trunc from 0x0286a3 to 0x02aeee (0x284c bytes) 000926 1788838555.630137 write 0x01fca4 thru 0x02895c (0x8cb9 bytes) 000927 1788838555.640493 write 0x0357af thru 0x0387b5 (0x3007 bytes) 000928 1788838555.646427 trunc from 0x040000 to 0x010ed5 (0x2f12a bytes) 000929 1788838555.672038 mapread 0x003fec thru 0x00ae2d (0x6e42 bytes) 000930 1788838555.694788 write 0x00e2da thru 0x0129ac (0x46d3 bytes) 000931 1788838555.700089 mapwrite 0x02ec49 thru 0x037ffb (0x93b3 bytes) 000932 1788838555.808411 fallocate 0x020837 thru 0x023849 (0x3013 bytes) 000933 1788838555.818558 trunc from 0x015747 to 0x019421 (0x3cdb bytes) 000934 1788838555.828358 mapwrite 0x03a3b1 thru 0x03ffff (0x5c4f bytes) 000935 1788838555.920599 mapwrite 0x03fa04 thru 0x03ffff (0x5fc bytes) 000936 1788838555.942896 trunc from 0x021697 to 0x02f82f (0xe199 bytes) 000937 1788838555.954459 mapread 0x013cc8 thru 0x017f58 (0x4291 bytes) 000938 1788838555.966351 trunc from 0x0167fb to 0x01bb4f (0x5355 bytes) 000939 1788838555.972571 write 0x020488 thru 0x02f851 (0xf3ca bytes) 000940 1788838555.992473 mapwrite 0x03a67c thru 0x03ba42 (0x13c7 bytes) 000941 1788838556.035851 mapwrite 0x0136d9 thru 0x019345 (0x5c6d bytes) 000942 1788838556.082513 punch from 0x020873 to 0x023f57 (0x36e5 bytes) 000943 1788838556.093210 mapwrite 0x023b72 thru 0x02af89 (0x7418 bytes) 000944 1788838556.132247 mapread 0x0347ec thru 0x03d234 (0x8a49 bytes) 000945 1788838556.134335 fallocate 0x02d1ec thru 0x033183 (0x5f98 bytes) 000946 1788838556.141073 read 0x03df29 thru 0x03ffff (0x20d7 bytes) 000947 1788838556.142121 mapwrite 0x022cba thru 0x0312d6 (0xe61d bytes) 000948 1788838556.163203 trunc from 0x040000 to 0x0372c6 (0x8d39 bytes) 000949 1788838556.187524 punch from 0x0051c7 to 0x006175 (0xfaf bytes) 000950 1788838556.198271 MIRROR_RESYNC 2 mirrors 000951 1788838557.300448 read 0x022f61 thru 0x03099b (0xda3b bytes) 000952 1788838557.311347 mapread 0x003d80 thru 0x00851d (0x479e bytes) 000953 1788838557.334148 trunc from 0x0372c7 to 0x012cb4 (0x24612 bytes) 000954 1788838557.381003 MIRROR_RESYNC 2 mirrors 000955 1788838558.404830 trunc from 0x012cb5 to 0x00225b (0x10a59 bytes) 000956 1788838558.486918 fallocate 0x03cef0 thru 0x03dcbd (0xdce bytes) 000957 1788838558.495311 write 0x00d39e thru 0x01626d (0x8ed0 bytes) 000958 1788838558.508413 mapread 0x032dbe thru 0x037512 (0x4755 bytes) 000959 1788838558.529469 write 0x003694 thru 0x00597f (0x22ec bytes) 000960 1788838558.543389 mapwrite 0x02ac7a thru 0x037dae (0xd135 bytes) 000961 1788838558.600558 read 0x0296a7 thru 0x035886 (0xc1e0 bytes) 000962 1788838558.601298 mapwrite 0x010367 thru 0x01bb89 (0xb823 bytes) 000963 1788838558.641133 fallocate 0x00d731 thru 0x01b295 (0xdb65 bytes) 000964 1788838558.646829 read 0x03b686 thru 0x03dcbd (0x2638 bytes) 000965 1788838558.657319 trunc from 0x00b268 to 0x0195d7 (0xe370 bytes) 000966 1788838558.663413 read 0x0023c2 thru 0x00d31f (0xaf5e bytes) 000967 1788838558.680564 read 0x001b87 thru 0x00aa95 (0x8f0f bytes) 000968 1788838558.682078 mapread 0x019e44 thru 0x020008 (0x61c5 bytes) 000969 1788838558.688366 fallocate 0x036b1f thru 0x03ffff (0x94e1 bytes) 000970 1788838558.694334 mapread 0x03d28a thru 0x03d9ef (0x766 bytes) 000971 1788838558.699508 trunc from 0x03afe6 to 0x03dcbd (0x2cd8 bytes) 000973 1788838558.716492 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15b:0x0]: Invalid argument (22) 000974 1788838558.912472 read 0x0353c8 thru 0x0397e2 (0x441b bytes) 000975 1788838558.927566 write 0x03ad30 thru 0x03d4e0 (0x27b1 bytes) 000976 1788838558.943066 mapread 0x0289ed thru 0x02cdce (0x43e2 bytes) 000978 1788838558.962563 read 0x00b31d thru 0x0119c6 (0x66aa bytes) 000979 1788838558.972897 trunc from 0x03dcbe to 0x02c8db (0x113e2 bytes) 000981 1788838559.014014 write 0x011fa5 thru 0x02035c (0xe3b8 bytes) 000982 1788838559.021080 write 0x020881 thru 0x026045 (0x57c5 bytes) 000983 1788838559.029200 MIRROR_EXTEND to 2 mirrors 000985 1788838560.090661 trunc from 0x02c8dc to 0x03e613 (0x11d38 bytes) 000986 1788838560.177764 mapread 0x024b38 thru 0x02a7d3 (0x5c9c bytes) 000987 1788838560.216754 trunc from 0x0008fc to 0x007340 (0x6a45 bytes) 000988 1788838560.235594 read 0x01a07e thru 0x0251b3 (0xb136 bytes) 000989 1788838560.250804 mapread 0x01826d thru 0x01cf64 (0x4cf8 bytes) 000990 1788838560.280589 write 0x02dcee thru 0x03d7e1 (0xfaf4 bytes) 000991 1788838560.304462 MIRROR_RESYNC 2 mirrors 000992 1788838561.465959 punch from 0x0095a6 to 0x009605 (0x60 bytes) 000993 1788838561.522684 mapwrite 0x02fa99 thru 0x03b5d7 (0xbb3f bytes) 000994 1788838561.595450 mapwrite 0x007ef9 thru 0x01799e (0xfaa6 bytes) 000995 1788838561.641986 mapread 0x0237a1 thru 0x032d32 (0xf592 bytes) 000996 1788838561.660298 mapread 0x01dadd thru 0x020c97 (0x31bb bytes) 000997 1788838561.661221 trunc from 0x0349e7 to 0x03e613 (0x9c2d bytes) 000998 1788838561.667424 fallocate 0x03f3e7 thru 0x03ffff (0xc19 bytes) 001000 1788838561.674657 write 0x0262fc thru 0x02e1bc (0x7ec1 bytes) All operations completed A-OK! PASS 70a (123s) == sanity-flr test 71: check mirror extend parent fid ==== 23:36:07 (1788838567) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x19b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 23:36:18 (1788838578) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.252472 s, 8.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913378 s, 11.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 23:36:28 (1788838588) Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre2 Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8856928 .. Extending file size to 5439392 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7987296 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8728640 .. Extending file size to 9279168 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8948576 .. Waiting 61219 61220 61222 61223 61227 done 192.168.204.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg443-client.virtnet /mnt/lustre2 (opts:) 192.168.204.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg443-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:37:54 (1788838674) Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre2 Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6763072 .. Extending file size to 8653280 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8194208 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9220000 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8202176 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8912320 .. Extending file size to 8948672 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62659 62660 62661 62663 62664 done 192.168.204.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg443-client.virtnet /mnt/lustre2 (opts:) 192.168.204.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg443-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:39:19 (1788838759) Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2196 1269108 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17860 3589160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25908 3581112 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43768 7170272 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2196 1269108 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17860 3589160 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25908 3581112 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43768 7170272 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108556 s, 19.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.239825 s, 8.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3a7433c3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg443-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:39:39 (1788838779) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197132 s, 10.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 23:39:47 (1788838787) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:39:55 (1788838795) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152426 s, 13.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:40:05 (1788838805) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167885 s, 12.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:40:13 (1788838813) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125865 s, 16.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:40:48 (1788838848) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg443-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100906 s, 10.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e8:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:41:45 (1788838905) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg443-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113739 s, 18.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:42:42 (1788838962) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg443-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg443-server: Pool lustre.test_204f created Adding targets to pool oleg443-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg443-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142884 s, 14.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg443-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg443-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg443-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:44:03 (1788839043) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:44:12 (1788839052) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:44:20 (1788839060) oleg443-server: Pool lustre.test_206 created Adding targets to pool oleg443-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg443-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg443-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg443-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg443-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:44:51 (1788839091) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108454 s, 9.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319986 s, 32.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf05aae76 Mirror 3: 0xf05aae76 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:45:03 (1788839103) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:45:08 (1788839108) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:45:13 (1788839113) Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0585243 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1080000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x17b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x17b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x17b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x870000 Read this (4 bytes): BBBB 192.168.204.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg443-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:45:55 (1788839155) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0279372 s, 293 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:46:02 (1788839162) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429738 s, 24.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x84 lmm_fid: [0x200000405:0x84:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611268 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:46:04.000000000 -0400 Modify: 2026-09-07 23:46:04.000000000 -0400 Change: 2026-09-07 23:46:05.000000000 -0400 Birth: 2026-09-07 23:46:04.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:46:11 (1788839171) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589885 s, 17.8 MB/s oleg443-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg443-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg443-server: O/280000401/d29/13d: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:46:37 (1788839197) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.341743 s, 120 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0377695 s, 108 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 23:46 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x89:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 23:46 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:46:47 (1788839207) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:46:57 (1788839217) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 1980 sec =========== 23:47:09 (1788839229) === sanity-flr: start cleanup 23:47:12 (1788839232) === === sanity-flr: finish cleanup 23:47:24 (1788839244) ===