-----============= acceptance-small: sanity-flr ============----- Sat Sep 12 22:07:49 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg407-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:08:11 (1789265291) === oleg407-client.virtnet: executing check_config_client /mnt/lustre oleg407-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg407-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8bdd8a283800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bdd8a283800.idle_timeout=debug disable quota as required oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:08:54 (1789265334) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8206] == sanity-flr test 0a: lfs mirror create with -N option == 22:08:56 (1789265336) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:09:13 (1789265353) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:09:16 (1789265356) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:09:19 (1789265359) PASS 0d (20s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:09:40 (1789265380) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:09:42 (1789265382) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 22:09:46 (1789265386) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8bdd8a283800.stats=clear osc.lustre-OST0001-osc-ffff8bdd8a283800.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:09:52 (1789265392) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:10:02 (1789265402) PASS 0j (6s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:10:08 (1789265408) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:10:12 (1789265412) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:10:19 (1789265419) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 22:10:28 (1789265428) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:10:35 (1789265435) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:10:42 (1789265442) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:10:49 (1789265449) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:10:59 (1789265459) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:11:08 (1789265468) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.58921 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124613 s, 8.4 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:11:18 (1789265478) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.387239 s, 37.9 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:11:26 (1789265486) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:11:35 (1789265495) Stopping /mnt/lustre-ost1 (opts:) on oleg407-server oleg407-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bdd8a283800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bdd8a283800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg407-server: oleg407-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg407-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg407-server oleg407-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bdd8a283800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bdd8a283800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg407-server: oleg407-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg407-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (61s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:12:36 (1789265556) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.494671 s, 17.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg407-server oleg407-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bdd8a283800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bdd8a283800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg407-server: oleg407-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg407-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:13:25 (1789265605) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg407-server oleg407-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bdd8a283800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bdd8a283800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg407-server: oleg407-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg407-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg407-server oleg407-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bdd8a283800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bdd8a283800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg407-server: oleg407-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg407-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:14:47 (1789265687) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.87628 s, 18.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 12 22:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.920s user 0m0.001s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg407-server oleg407-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bdd8a283800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bdd8a283800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.375s user 0m0.001s sys 0m0.171s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg407-server: oleg407-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg407-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg407-server oleg407-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bdd8a283800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bdd8a283800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.061s user 0m0.001s sys 0m0.176s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg407-server: oleg407-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg407-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg407-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bdd8a283800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bdd8a283800.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:16:19 (1789265779) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:16:23 (1789265783) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:16:26 (1789265786) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 22:16:30 (1789265790) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 22:16:38 (1789265798) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.405166 s, 31.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:16:51 (1789265811) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283543 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418112 s, 25.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:16:58 (1789265818) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:17:08 (1789265828) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109931 s, 931 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00272877 s, 375 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00252892 s, 405 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00326604 s, 314 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 22:17:18 (1789265838) create test files with size 15729580 .. dec1efe0e75fe95cba9cbb95062ea765 - 1d9165ac54010c5cad48f943601a17b6 - 5c0f5e6370e19d9b0c4e4a3688da3253 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg407-client.virtnet /mnt/lustre (opts:) Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 22:17:43 (1789265863) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:17:56 (1789265876) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:18:02 (1789265882) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657134 s, 16.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276082 s, 38.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 22:18:10 (1789265890) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.059235 s, 35.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.375 s, 11.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121777 s, 17.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 22:18:25 (1789265905) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:18:28 (1789265908) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:18:32 (1789265912) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307217 s, 34.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 22:18:39 (1789265919) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.237996 s, 13.2 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:18:48 (1789265928) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:18:58 (1789265938) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.322341 s, 32.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg407-client.virtnet /mnt/lustre (opts:) Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:19:09 (1789265949) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8bdd916f4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.450601 s, 23.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg407-client.virtnet /mnt/lustre (opts:) Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8bdd89fa9800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:19:24 (1789265964) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:20:05 (1789266005) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:20:43 (1789266043) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:20:46 (1789266046) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.255202 s, 12.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.187326 s, 16.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x57ad7923 (nosync,stale) Mirror 2: 0x57ad7923 (nosync,stale) ** verify mirror 2 doesn't change original checksum: ac002662b18d757eb51c1c4e0b5b5139 - mirror 1 checksum: a49b605b1d1a881a86de759fa4e9db5c - mirror 2 checksum: ac002662b18d757eb51c1c4e0b5b5139 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-12T22:20:49-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-12T22:20:49-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: ac002662b18d757eb51c1c4e0b5b5139 - mirror 1 checksum: a49b605b1d1a881a86de759fa4e9db5c - mirror 2 checksum: a49b605b1d1a881a86de759fa4e9db5c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-12T22:20:49-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-12T22:20:49-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:20:59 (1789266059) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822403 s, 12.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:21:08 (1789266068) Failing mds1 on oleg407-server Stopping /mnt/lustre-mds1 (opts:) on oleg407-server 22:21:21 (1789266081) shut down facet: mds1 facet_host: oleg407-server facet_failover_host: oleg407-server Failover mds1 to oleg407-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Started lustre-MDT0000 22:21:45 (1789266105) targets are mounted 22:21:45 (1789266105) facet_failover done oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650606 s, 16.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:22:01 (1789266121) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0386588 s, 530 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0271688 s, 754 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:22:11 (1789266131) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0504165 s, 406 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00867538 s, 2.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:22:22 (1789266142) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0209014 s, 784 kB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:22:30 (1789266150) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0508378 s, 403 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:22:37 (1789266157) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00524658 s, 781 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0287156 s, 143 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:22:45 (1789266165) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:23:04 (1789266184) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:23:31 (1789266211) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:23:55 (1789266235) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:24:19 (1789266259) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 22:24:26 (1789266266) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57552 mirror split pid 57554 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57552 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57554 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 22:25:34 (1789266334) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 6337 skipping zero length zero range fallocating to largest ever: 0x109a6 000003 1789266338.201371 fallocate 0x00fa1d thru 0x0109a5 (0xf89 bytes) 000004 1789266338.209016 write 0x01b120 thru 0x02270a (0x75eb bytes) 000005 1789266338.210665 mapread 0x00ab09 thru 0x00e90b (0x3e03 bytes) truncating to largest ever: 0x2a67e 000006 1789266338.248945 trunc from 0x02270b to 0x02a67d (0x7f73 bytes) 000007 1789266338.270045 trunc from 0x0296e7 to 0x02a67d (0xf97 bytes) truncating to largest ever: 0x36c20 000008 1789266338.276068 trunc from 0x02a67e to 0x036c1f (0xc5a2 bytes) fallocating to largest ever: 0x40000 000009 1789266338.307513 fallocate 0x038c72 thru 0x03ffff (0x738e bytes) 000010 1789266338.311419 write 0x01ba7f thru 0x02b80e (0xfd90 bytes) 000011 1789266338.315842 write 0x033b33 thru 0x03f17f (0xb64d bytes) 000012 1789266338.324076 mapwrite 0x02ac48 thru 0x037fe5 (0xd39e bytes) 000013 1789266338.359974 fallocate 0x0238b9 thru 0x03273c (0xee84 bytes) 000014 1789266338.365666 punch from 0x01ee28 to 0x020d8e (0x1f67 bytes) 000018 1789266338.375653 write 0x03f206 thru 0x03f37e (0x179 bytes) 000019 1789266338.376688 read 0x03aacc thru 0x03ffff (0x5534 bytes) 000020 1789266338.377255 read 0x01cf47 thru 0x02885e (0xb918 bytes) 000021 1789266338.383858 fallocate 0x027412 thru 0x027449 (0x38 bytes) 000023 1789266338.389131 punch from 0x008bbc to 0x015dd6 (0xd21b bytes) 000024 1789266338.393393 read 0x022a7a thru 0x024b61 (0x20e8 bytes) 000025 1789266338.398257 MIRROR_EXTEND to 2 mirrors 000026 1789266339.105774 punch from 0x02fd8c to 0x0342d7 (0x454c bytes) 000027 1789266339.143315 trunc from 0x040000 to 0x02be69 (0x14196 bytes) 000028 1789266339.201571 write 0x0093c5 thru 0x018cb9 (0xf8f5 bytes) 000029 1789266339.221014 fallocate 0x008095 thru 0x00c8dd (0x4849 bytes) 000030 1789266339.226627 write 0x0382c6 thru 0x03ffff (0x7d3a bytes) 000031 1789266339.232387 fallocate 0x02a052 thru 0x032b54 (0x8b03 bytes) 000033 1789266339.240596 punch from 0x02dff9 to 0x03b22a (0xd232 bytes) 000034 1789266339.265951 mapread 0x0151ca thru 0x020d8d (0xbbc4 bytes) 000035 1789266339.302840 mapread 0x032554 thru 0x039b32 (0x75df bytes) 000036 1789266339.318012 fallocate 0x00c2e8 thru 0x010368 (0x4081 bytes) 000038 1789266339.324990 fallocate 0x020316 thru 0x027c82 (0x796d bytes) 000040 1789266339.334933 write 0x0198ff thru 0x01e708 (0x4e0a bytes) 000041 1789266339.336487 fallocate 0x025adb thru 0x0302db (0xa801 bytes) 000042 1789266339.344524 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) 000043 1789266339.508123 punch from 0x010ee5 to 0x01f801 (0xe91d bytes) 000045 1789266339.518404 punch from 0x000873 to 0x00e5c1 (0xdd4f bytes) 000046 1789266339.524303 read 0x020781 thru 0x028e74 (0x86f4 bytes) 000048 1789266339.539246 trunc from 0x040000 to 0x009f16 (0x360e9 bytes) 000050 1789266339.565199 trunc from 0x009f17 to 0x002226 (0x7cf0 bytes) 000052 1789266339.584255 mapread 0x001fa8 thru 0x002226 (0x27f bytes) 000053 1789266339.600261 MIRROR_EXTEND to 2 mirrors 000054 1789266340.118716 write 0x0005e7 thru 0x004a15 (0x442f bytes) 000055 1789266340.172464 fallocate 0x01c6ec thru 0x01f225 (0x2b3a bytes) 000056 1789266340.178727 mapwrite 0x008d84 thru 0x00a3e3 (0x1660 bytes) 000057 1789266340.278493 read 0x004d38 thru 0x00a3e3 (0x56ac bytes) 000058 1789266340.279583 trunc from 0x00a3e4 to 0x0218a2 (0x174bf bytes) 000059 1789266340.295450 trunc from 0x0218a3 to 0x0229e6 (0x1144 bytes) 000060 1789266340.310464 punch from 0x013c1d to 0x01db87 (0x9f6b bytes) 000061 1789266340.315702 trunc from 0x011559 to 0x012fbd (0x1a65 bytes) 000062 1789266340.320410 mapread 0x005846 thru 0x0092c4 (0x3a7f bytes) 000063 1789266340.321574 mapread 0x016385 thru 0x0229e6 (0xc662 bytes) 000064 1789266340.335812 punch from 0x00d53b to 0x011b1f (0x45e5 bytes) 000065 1789266340.341493 punch from 0x01eb81 to 0x0229e6 (0x3e66 bytes) 000066 1789266340.345994 mapread 0x005e50 thru 0x012418 (0xc5c9 bytes) 000067 1789266340.355917 read 0x001d59 thru 0x002b00 (0xda8 bytes) 000068 1789266340.356589 fallocate 0x02fddf thru 0x0321aa (0x23cc bytes) 000069 1789266340.361465 write 0x034ae0 thru 0x03cb7a (0x809b bytes) 000070 1789266340.366463 mapwrite 0x015dd4 thru 0x025d60 (0xff8d bytes) 000071 1789266340.404194 read 0x017b1a thru 0x01edc6 (0x72ad bytes) 000072 1789266340.404901 mapwrite 0x025db1 thru 0x0339a9 (0xdbf9 bytes) 000073 1789266340.428077 write 0x00cb48 thru 0x019b53 (0xd00c bytes) 000074 1789266340.430519 mapread 0x0360c0 thru 0x03cb7a (0x6abb bytes) 000075 1789266340.431849 read 0x022be9 thru 0x0283dd (0x57f5 bytes) 000077 1789266340.433005 trunc from 0x03cb7b to 0x0032c6 (0x398b4 bytes) 000079 1789266340.451328 read 0x001822 thru 0x00315a (0x1939 bytes) 000080 1789266340.452327 mapwrite 0x00ae55 thru 0x018ed4 (0xe080 bytes) 000081 1789266340.501387 trunc from 0x0104c1 to 0x018ed4 (0x8a14 bytes) 000082 1789266340.511188 trunc from 0x00cfb2 to 0x010582 (0x35d1 bytes) 000084 1789266340.521100 MIRROR_RESYNC 2 mirrors 000085 1789266341.164853 trunc from 0x018ed5 to 0x019cb0 (0xddc bytes) 000086 1789266341.249153 write 0x0389f6 thru 0x03db6c (0x5177 bytes) 000087 1789266341.251458 punch from 0x0334d7 to 0x03d704 (0xa22e bytes) 000088 1789266341.267961 mapwrite 0x01f7f1 thru 0x02cf69 (0xd779 bytes) 000089 1789266341.330044 trunc from 0x008695 to 0x00d0f8 (0x4a64 bytes) 000090 1789266341.336313 mapread 0x037b45 thru 0x03db6c (0x6028 bytes) 000091 1789266341.360903 trunc from 0x03db6d to 0x030409 (0xd763 bytes) 000092 1789266341.400481 fallocate 0x00a083 thru 0x013723 (0x96a1 bytes) 000093 1789266341.418005 write 0x005e61 thru 0x011018 (0xb1b8 bytes) 000094 1789266341.438510 punch from 0x01592e to 0x015b89 (0x25c bytes) 000095 1789266341.454738 trunc from 0x008edd to 0x012bb7 (0x9cdb bytes) 000096 1789266341.475017 write 0x033893 thru 0x035a20 (0x218e bytes) 000097 1789266341.477144 write 0x026399 thru 0x02a1eb (0x3e53 bytes) 000098 1789266341.482862 fallocate 0x0264d4 thru 0x035c6a (0xf797 bytes) 000099 1789266341.487873 punch from 0x002622 to 0x00d582 (0xaf61 bytes) 000100 1789266341.499175 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) 000101 1789266341.639323 trunc from 0x035a21 to 0x02e799 (0x7287 bytes) 000103 1789266341.672031 punch from 0x02d0fe to 0x02e799 (0x169c bytes) 000104 1789266341.684446 read 0x02b65a thru 0x02e799 (0x3140 bytes) 000105 1789266341.691658 mapread 0x022322 thru 0x02b7fd (0x94dc bytes) 000107 1789266341.718975 write 0x021024 thru 0x02961e (0x85fb bytes) 000108 1789266341.720869 punch from 0x01c947 to 0x020d69 (0x4423 bytes) 000109 1789266341.725925 mapread 0x0120af thru 0x01b0ad (0x8fff bytes) 000110 1789266341.753696 read 0x0107a7 thru 0x015795 (0x4fef bytes) 000111 1789266341.754504 trunc from 0x01e62d to 0x02e049 (0xfa1d bytes) 000112 1789266341.768937 mapwrite 0x01bcd9 thru 0x021c0c (0x5f34 bytes) 000114 1789266341.798946 punch from 0x023342 to 0x02e2b1 (0xaf70 bytes) 000115 1789266341.804087 mapwrite 0x004f2a thru 0x007474 (0x254b bytes) 000116 1789266341.828161 read 0x010862 thru 0x01ac46 (0xa3e5 bytes) 000117 1789266341.828804 fallocate 0x0060ec thru 0x010afc (0xaa11 bytes) 000119 1789266341.843687 MIRROR_EXTEND to 2 mirrors 000120 1789266342.868005 read 0x016402 thru 0x017bef (0x17ee bytes) 000121 1789266342.887533 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) 000122 1789266343.020176 punch from 0x002cda to 0x00c083 (0x93aa bytes) 000123 1789266343.037754 punch from 0x001121 to 0x00f701 (0xe5e1 bytes) 000125 1789266343.046527 MIRROR_EXTEND to 2 mirrors 000126 1789266343.752930 fallocate 0x02f901 thru 0x039a4d (0xa14d bytes) 000127 1789266343.793907 punch from 0x005f78 to 0x008db5 (0x2e3e bytes) 000128 1789266343.807776 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2e79a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e79a): Mirror 2: 0x85a4cb49 (stale) CRC-32 checksum value for chunk [0, 0x2e79a): Mirror 1: 0x85a4cb49 Mirror 3: 0x85a4cb49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e79a: skipped 000129 1789266344.696482 punch from 0x02ae27 to 0x02e799 (0x3973 bytes) 000130 1789266344.732164 fallocate 0x0110a3 thru 0x014460 (0x33be bytes) 000131 1789266344.740744 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2e79a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e79a): Mirror 2: 0x85a4cb49 (stale) Mirror 3: 0x85a4cb49 (stale) CRC-32 checksum value for chunk [0, 0x2e79a): Mirror 1: 0x85a4cb49 Mirror 4: 0x85a4cb49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e79a: skipped 000132 1789266345.758934 write 0x02bdb3 thru 0x02e582 (0x27d0 bytes) 000133 1789266345.819731 mapwrite 0x01ef0f thru 0x02d427 (0xe519 bytes) 000134 1789266345.886036 read 0x014a61 thru 0x022dbb (0xe35b bytes) 000135 1789266345.896270 trunc from 0x02e79a to 0x036af9 (0x8360 bytes) 000136 1789266345.925081 read 0x021a95 thru 0x025fd0 (0x453c bytes) 000138 1789266345.927315 fallocate 0x00c73d thru 0x019a58 (0xd31c bytes) 000141 1789266345.935787 read 0x0327d6 thru 0x036af9 (0x4324 bytes) 000142 1789266345.954166 fallocate 0x024f10 thru 0x0321d6 (0xd2c7 bytes) 000143 1789266345.963535 trunc from 0x006050 to 0x007dea (0x1d9b bytes) 000144 1789266345.968638 mapwrite 0x016040 thru 0x019a60 (0x3a21 bytes) 000145 1789266345.987616 write 0x03afae thru 0x03ffff (0x5052 bytes) 000146 1789266345.991585 mapwrite 0x00b654 thru 0x0162cd (0xac7a bytes) 000147 1789266346.023275 fallocate 0x03ab47 thru 0x03c8b8 (0x1d72 bytes) 000148 1789266346.036175 trunc from 0x040000 to 0x007b68 (0x38497 bytes) 000149 1789266346.077085 trunc from 0x007b69 to 0x00061a (0x754e bytes) 000150 1789266346.093615 mapwrite 0x010300 thru 0x01d33c (0xd03d bytes) 000153 1789266346.228263 read 0x002a51 thru 0x00b658 (0x8c08 bytes) 000154 1789266346.241731 fallocate 0x012fd3 thru 0x015f73 (0x2fa1 bytes) 000155 1789266346.257150 read 0x00f1ce thru 0x014598 (0x53cb bytes) 000156 1789266346.257865 trunc from 0x01d33d to 0x008792 (0x14baa bytes) 000157 1789266346.282362 mapwrite 0x02a441 thru 0x031389 (0x6f49 bytes) 000158 1789266346.325273 mapread 0x02a021 thru 0x031389 (0x7369 bytes) 000159 1789266346.326574 mapread 0x01da98 thru 0x01de7a (0x3e3 bytes) 000160 1789266346.328146 punch from 0x0245c2 to 0x024aa7 (0x4e6 bytes) 000161 1789266346.333948 trunc from 0x03138a to 0x014ae3 (0x1c8a6 bytes) 000162 1789266346.369126 punch from 0x00336f to 0x00dec1 (0xab53 bytes) 000163 1789266346.376565 read 0x00196c thru 0x00dd34 (0xc3c9 bytes) 000165 1789266346.397153 trunc from 0x00888f to 0x012c07 (0xa379 bytes) 000166 1789266346.413803 MIRROR_RESYNC 4 mirrors 000167 1789266347.102494 write 0x01a811 thru 0x0297ae (0xef9e bytes) 000168 1789266347.137173 trunc from 0x01ca2a to 0x01f3b3 (0x298a bytes) 000169 1789266347.158290 trunc from 0x018b67 to 0x0265b5 (0xda4f bytes) 000170 1789266347.166421 trunc from 0x0297af to 0x0228d8 (0x6ed6 bytes) 000171 1789266347.226693 read 0x021071 thru 0x022784 (0x1714 bytes) 000172 1789266347.237536 fallocate 0x008ce0 thru 0x012151 (0x9472 bytes) 000173 1789266347.242216 mapread 0x01a90c thru 0x01ab25 (0x21a bytes) 000174 1789266347.248596 read 0x020580 thru 0x0228d8 (0x2359 bytes) 000175 1789266347.267738 read 0x01829a thru 0x01adae (0x2b15 bytes) 000176 1789266347.268388 trunc from 0x0228d9 to 0x01effd (0x38db bytes) 000177 1789266347.290929 trunc from 0x0131cc to 0x013a66 (0x89b bytes) 000178 1789266347.303791 trunc from 0x01effe to 0x01ef4b (0xb2 bytes) 000179 1789266347.336801 MIRROR_RESYNC 4 mirrors 000180 1789266348.208507 MIRROR_EXTEND to 5 mirrors 000181 1789266348.882798 mapwrite 0x01a966 thru 0x01c777 (0x1e12 bytes) 000182 1789266348.988926 write 0x034ba2 thru 0x03ffff (0xb45e bytes) 000183 1789266348.996516 fallocate 0x03bd47 thru 0x03ffff (0x42b9 bytes) 000184 1789266349.001015 mapwrite 0x016829 thru 0x01d569 (0x6d41 bytes) 000185 1789266349.026707 punch from 0x037101 to 0x03e0d6 (0x6fd6 bytes) 000186 1789266349.035660 mapwrite 0x021c91 thru 0x0261e5 (0x4555 bytes) 000187 1789266349.077887 mapread 0x017735 thru 0x0202e2 (0x8bae bytes) 000188 1789266349.080996 read 0x01248c thru 0x018d46 (0x68bb bytes) 000189 1789266349.090232 mapread 0x02c29a thru 0x03b5a8 (0xf30f bytes) 000190 1789266349.133599 MIRROR_RESYNC 5 mirrors 000191 1789266350.692080 MIRROR_EXTEND to 6 mirrors 000192 1789266351.683211 trunc from 0x040000 to 0x000fa0 (0x3f05f bytes) 000193 1789266351.743538 fallocate 0x010a11 thru 0x018941 (0x7f31 bytes) 000194 1789266351.753403 trunc from 0x00036f to 0x000fa0 (0xc32 bytes) 000195 1789266351.758436 punch from 0x000f8f to 0x000fa0 (0x12 bytes) 000196 1789266351.764591 punch from 0x000b99 to 0x000fa0 (0x408 bytes) 000198 1789266351.773156 read 0x000520 thru 0x000fa0 (0xa81 bytes) 000199 1789266351.781286 write 0x032359 thru 0x03f910 (0xd5b8 bytes) 000200 1789266351.784820 read 0x03de8f thru 0x03f910 (0x1a82 bytes) 000201 1789266351.786077 mapread 0x02da6e thru 0x0316d5 (0x3c68 bytes) 000202 1789266351.796840 read 0x019256 thru 0x02615b (0xcf06 bytes) 000203 1789266351.798766 punch from 0x012d2a to 0x01868a (0x5961 bytes) 000204 1789266351.803553 punch from 0x027f7e to 0x03755e (0xf5e1 bytes) 000205 1789266351.820586 write 0x013f94 thru 0x01811d (0x418a bytes) 000206 1789266351.848103 read 0x01f64a thru 0x02d5d7 (0xdf8e bytes) 000207 1789266351.849165 read 0x02bf06 thru 0x031ac8 (0x5bc3 bytes) 000208 1789266351.849709 read 0x029e40 thru 0x02dab4 (0x3c75 bytes) 000209 1789266351.851314 fallocate 0x01db12 thru 0x02995c (0xbe4b bytes) 000210 1789266351.865414 read 0x0222fa thru 0x0262fd (0x4004 bytes) 000211 1789266351.868083 fallocate 0x024d1c thru 0x02c2c0 (0x75a5 bytes) 000212 1789266351.880920 mapread 0x02ea51 thru 0x03e0e7 (0xf697 bytes) 000213 1789266351.883169 mapread 0x01e694 thru 0x02665f (0x7fcc bytes) 000215 1789266351.887090 read 0x02190f thru 0x028dfe (0x74f0 bytes) 000216 1789266351.888122 trunc from 0x01d48a to 0x021ae6 (0x465d bytes) 000217 1789266351.892227 punch from 0x01e13e to 0x022cb3 (0x4b76 bytes) 000218 1789266351.897288 mapwrite 0x0362b9 thru 0x03ffff (0x9d47 bytes) 000219 1789266351.947801 fallocate 0x02da52 thru 0x0360a8 (0x8657 bytes) 000220 1789266351.954692 punch from 0x00b784 to 0x00d155 (0x19d2 bytes) 000221 1789266351.966010 trunc from 0x02db76 to 0x03cdf7 (0xf282 bytes) 000222 1789266351.977449 mapread 0x02acc1 thru 0x03691c (0xbc5c bytes) 000223 1789266351.997524 read 0x02adcc thru 0x02b09b (0x2d0 bytes) 000224 1789266352.005216 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x998f8401 (stale) Mirror 3: 0x998f8401 (stale) Mirror 4: 0x998f8401 (stale) Mirror 5: 0x998f8401 (stale) Mirror 6: 0x998f8401 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7ae77925 Mirror 7: 0x7ae77925 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000225 1789266352.972117 trunc from 0x040000 to 0x01f78e (0x20871 bytes) 000227 1789266353.045658 punch from 0x014094 to 0x016c80 (0x2bed bytes) 000228 1789266353.057655 write 0x025252 thru 0x02dc39 (0x89e8 bytes) 000230 1789266353.061610 mapread 0x00a172 thru 0x00b827 (0x16b6 bytes) 000231 1789266353.074221 fallocate 0x00be0d thru 0x0128e9 (0x6add bytes) 000232 1789266353.080606 fallocate 0x004313 thru 0x0138e5 (0xf5d3 bytes) 000233 1789266353.086587 mapread 0x0283a8 thru 0x02dc39 (0x5892 bytes) 000234 1789266353.087986 punch from 0x005a47 to 0x00d15a (0x7714 bytes) 000235 1789266353.093732 fallocate 0x012fae thru 0x012fbc (0xf bytes) 000237 1789266353.101206 trunc from 0x02dc3a to 0x03a9c2 (0xcd89 bytes) 000238 1789266353.128293 mapread 0x0213b9 thru 0x02e883 (0xd4cb bytes) 000239 1789266353.150600 write 0x0064e3 thru 0x00f91e (0x943c bytes) 000240 1789266353.163659 trunc from 0x01a3ef to 0x02106c (0x6c7e bytes) 000241 1789266353.172231 trunc from 0x0321c7 to 0x036d28 (0x4b62 bytes) 000242 1789266353.186705 trunc from 0x031050 to 0x0391f5 (0x81a6 bytes) 000243 1789266353.201176 trunc from 0x03a9c3 to 0x00685d (0x34165 bytes) 000244 1789266353.235614 read 0x0004da thru 0x001aad (0x15d4 bytes) 000245 1789266353.247232 read 0x005de7 thru 0x00685d (0xa77 bytes) 000246 1789266353.259297 mapwrite 0x005c07 thru 0x00dfb5 (0x83af bytes) 000247 1789266353.335854 mapwrite 0x022853 thru 0x02cec6 (0xa674 bytes) 000248 1789266353.458651 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2cec7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cec7): Mirror 2: 0xf46aa396 (stale) Mirror 3: 0xf46aa396 (stale) Mirror 4: 0xf46aa396 (stale) Mirror 5: 0xf46aa396 (stale) Mirror 6: 0xf46aa396 (stale) Mirror 7: 0x8b189c10 (stale) CRC-32 checksum value for chunk [0, 0x2cec7): Mirror 1: 0x7180d02 Mirror 8: 0x7180d02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cec7: skipped 000250 1789266354.886893 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2cec7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cec7): Mirror 2: 0xf46aa396 (stale) Mirror 3: 0xf46aa396 (stale) Mirror 4: 0xf46aa396 (stale) Mirror 5: 0xf46aa396 (stale) Mirror 6: 0xf46aa396 (stale) Mirror 7: 0x8b189c10 (stale) CRC-32 checksum value for chunk [0, 0x2cec7): Mirror 1: 0x7180d02 Mirror 8: 0x7180d02 Mirror 9: 0x7180d02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cec7: skipped 000251 1789266356.136923 fallocate 0x017184 thru 0x0248c3 (0xd740 bytes) 000252 1789266356.188781 read 0x022720 thru 0x02cec6 (0xa7a7 bytes) 000253 1789266356.200217 read 0x01c226 thru 0x02b435 (0xf210 bytes) 000254 1789266356.213880 MIRROR_RESYNC 9 mirrors 000255 1789266358.298434 mapwrite 0x0099d8 thru 0x00aad2 (0x10fb bytes) 000256 1789266358.406385 fallocate 0x039aa0 thru 0x03f21d (0x577e bytes) 000257 1789266358.412514 read 0x005011 thru 0x0144b8 (0xf4a8 bytes) 000259 1789266358.431737 mapread 0x00a4f3 thru 0x011aea (0x75f8 bytes) 000260 1789266358.432845 fallocate 0x038b60 thru 0x039f08 (0x13a9 bytes) 000261 1789266358.440874 read 0x015d1a thru 0x01b23f (0x5526 bytes) 000262 1789266358.442877 trunc from 0x00b45c to 0x00ce9c (0x1a41 bytes) 000263 1789266358.451008 write 0x035714 thru 0x03ffff (0xa8ec bytes) 000264 1789266358.454097 mapwrite 0x021996 thru 0x02f32f (0xd99a bytes) 000265 1789266358.486020 mapread 0x0362a5 thru 0x03fb9c (0x98f8 bytes) 000266 1789266358.487754 mapwrite 0x022632 thru 0x02f288 (0xcc57 bytes) 000267 1789266358.514626 trunc from 0x02d230 to 0x03d050 (0xfe21 bytes) 000268 1789266358.519715 trunc from 0x040000 to 0x00a60a (0x359f5 bytes) 000269 1789266358.575414 trunc from 0x00a60b to 0x0026fc (0x7f0e bytes) 000270 1789266358.604078 trunc from 0x001afa to 0x0026fc (0xc03 bytes) 000271 1789266358.613746 mapwrite 0x007d09 thru 0x009a71 (0x1d69 bytes) 000272 1789266358.677222 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9a72): Mirror 2: 0x3b0b4093 (stale) Mirror 3: 0x3b0b4093 (stale) Mirror 4: 0x3b0b4093 (stale) Mirror 5: 0x3b0b4093 (stale) Mirror 6: 0x3b0b4093 (stale) Mirror 7: 0x3b0b4093 (stale) Mirror 9: 0x3b0b4093 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9a72): Mirror 2: 0x3b0b4093 (stale) Mirror 3: 0x3b0b4093 (stale) Mirror 4: 0x3b0b4093 (stale) Mirror 5: 0x3b0b4093 (stale) Mirror 6: 0x3b0b4093 (stale) Mirror 7: 0x3b0b4093 (stale) Mirror 9: 0x3b0b4093 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9a72: skipped 000273 1789266359.200494 punch from 0x008c60 to 0x009a71 (0xe12 bytes) 000274 1789266359.212684 mapread 0x008832 thru 0x009a71 (0x1240 bytes) 000276 1789266359.225225 trunc from 0x002dae to 0x009a71 (0x6cc4 bytes) 000277 1789266359.235568 punch from 0x005caf to 0x009a71 (0x3dc3 bytes) 000279 1789266359.253163 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9a72): Mirror 2: 0x3b0b4093 (stale) Mirror 3: 0x3b0b4093 (stale) Mirror 4: 0x3b0b4093 (stale) Mirror 5: 0x3b0b4093 (stale) Mirror 6: 0x3b0b4093 (stale) Mirror 9: 0x3b0b4093 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9a72): Mirror 2: 0x3b0b4093 (stale) Mirror 3: 0x3b0b4093 (stale) Mirror 4: 0x3b0b4093 (stale) Mirror 5: 0x3b0b4093 (stale) Mirror 6: 0x3b0b4093 (stale) Mirror 9: 0x3b0b4093 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9a72: skipped 000280 1789266359.644314 read 0x00268d thru 0x009a71 (0x73e5 bytes) 000281 1789266359.662497 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x9a72): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9a72): Mirror 2: 0x3b0b4093 (stale) Mirror 3: 0x3b0b4093 (stale) Mirror 4: 0x3b0b4093 (stale) Mirror 5: 0x3b0b4093 (stale) Mirror 6: 0x3b0b4093 (stale) Mirror 9: 0x3b0b4093 (stale) CRC-32 checksum value for chunk [0, 0x9a72): Mirror 1: 0xe6895bc4 Mirror 10: 0xe6895bc4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9a72: skipped 000282 1789266360.585500 mapwrite 0x0348ad thru 0x03ffff (0xb753 bytes) 000283 1789266360.765839 mapwrite 0x00624b thru 0x007c1a (0x19d0 bytes) 000284 1789266360.795978 fallocate 0x00fdca thru 0x016364 (0x659b bytes) 000285 1789266360.808471 fallocate 0x01a19c thru 0x01ebed (0x4a52 bytes) 000286 1789266360.819481 mapread 0x023029 thru 0x025bdf (0x2bb7 bytes) 000287 1789266360.835139 fallocate 0x00859b thru 0x00ea62 (0x64c8 bytes) 000288 1789266360.848029 write 0x006b27 thru 0x00cffc (0x64d6 bytes) 000289 1789266360.854989 mapread 0x01a5f9 thru 0x026abd (0xc4c5 bytes) 000290 1789266360.896724 trunc from 0x040000 to 0x022b31 (0x1d4ce bytes) 000291 1789266360.952792 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x22b32): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22b32): Mirror 2: 0xd40b0aee (stale) Mirror 3: 0xd40b0aee (stale) Mirror 4: 0xd40b0aee (stale) Mirror 5: 0xd40b0aee (stale) Mirror 6: 0xd40b0aee (stale) Mirror 9: 0xd40b0aee (stale) Mirror 10: 0x9daa582 (stale) CRC-32 checksum value for chunk [0, 0x22b32): Mirror 1: 0xa0aacf99 Mirror 11: 0xa0aacf99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22b32: skipped 000293 1789266362.092315 read 0x0176bb thru 0x017eee (0x834 bytes) 000294 1789266362.133301 trunc from 0x00d1ab to 0x00fe6c (0x2cc2 bytes) 000297 1789266362.259298 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22b32): Mirror 2: 0xd40b0aee (stale) Mirror 4: 0xd40b0aee (stale) Mirror 5: 0xd40b0aee (stale) Mirror 6: 0xd40b0aee (stale) Mirror 9: 0xd40b0aee (stale) Mirror 10: 0x9daa582 (stale) Mirror 11: 0xa0aacf99 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22b32): Mirror 2: 0xd40b0aee (stale) Mirror 4: 0xd40b0aee (stale) Mirror 5: 0xd40b0aee (stale) Mirror 6: 0xd40b0aee (stale) Mirror 9: 0xd40b0aee (stale) Mirror 10: 0x9daa582 (stale) Mirror 11: 0xa0aacf99 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22b32: skipped 000298 1789266362.869748 mapwrite 0x02fcd8 thru 0x03ad24 (0xb04d bytes) 000299 1789266362.951928 trunc from 0x03ad25 to 0x01be1e (0x1ef06 bytes) 000300 1789266362.980928 mapwrite 0x03f734 thru 0x03ffff (0x8cc bytes) 000301 1789266363.024022 punch from 0x03b618 to 0x03db2b (0x2514 bytes) 000302 1789266363.031539 read 0x01f4aa thru 0x027f94 (0x8aeb bytes) 000303 1789266363.041592 read 0x022c97 thru 0x02a522 (0x788c bytes) 000304 1789266363.050427 fallocate 0x004e47 thru 0x00b1fa (0x63b4 bytes) 000305 1789266363.055849 fallocate 0x021ebb thru 0x02b4e2 (0x9628 bytes) 000306 1789266363.062794 punch from 0x01084c to 0x016055 (0x580a bytes) 000307 1789266363.068546 fallocate 0x0235b9 thru 0x0295e8 (0x6030 bytes) 000308 1789266363.073928 trunc from 0x0339e4 to 0x03900a (0x5627 bytes) 000310 1789266363.080528 write 0x02304b thru 0x0277d6 (0x478c bytes) 000312 1789266363.082800 write 0x000be6 thru 0x00f144 (0xe55f bytes) 000313 1789266363.084434 read 0x038c22 thru 0x03ffff (0x73de bytes) 000314 1789266363.094545 mapwrite 0x010219 thru 0x019772 (0x955a bytes) 000315 1789266363.119013 fallocate 0x008a06 thru 0x01738a (0xe985 bytes) 000317 1789266363.124575 punch from 0x015522 to 0x023083 (0xdb62 bytes) 000318 1789266363.130326 punch from 0x033444 to 0x03ffff (0xcbbc bytes) 000319 1789266363.139733 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x71ad765d (stale) Mirror 4: 0x71ad765d (stale) Mirror 5: 0x71ad765d (stale) Mirror 6: 0x71ad765d (stale) Mirror 9: 0x71ad765d (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0x4aa03f91 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcae06c14 Mirror 12: 0xcae06c14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000320 1789266364.125552 read 0x017d7e thru 0x01fe5e (0x80e1 bytes) 000321 1789266364.141845 mapwrite 0x03ac6c thru 0x03dc8e (0x3023 bytes) 000322 1789266364.281975 fallocate 0x03cbb2 thru 0x03ffff (0x344e bytes) 000323 1789266364.289319 trunc from 0x040000 to 0x005528 (0x3aad7 bytes) 000324 1789266364.420107 trunc from 0x005529 to 0x01a05c (0x14b34 bytes) 000325 1789266364.449884 write 0x001c1c thru 0x0113c8 (0xf7ad bytes) 000326 1789266364.460324 fallocate 0x033a52 thru 0x036ff3 (0x35a2 bytes) 000327 1789266364.469842 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1a05d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a05d): Mirror 2: 0x4995f277 (stale) Mirror 4: 0x4995f277 (stale) Mirror 5: 0x4995f277 (stale) Mirror 6: 0x4995f277 (stale) Mirror 9: 0x4995f277 (stale) Mirror 10: 0x8e444382 (stale) Mirror 11: 0x3649df28 (stale) Mirror 12: 0x10d610cf (stale) CRC-32 checksum value for chunk [0, 0x1a05d): Mirror 1: 0xd7940994 Mirror 13: 0xd7940994 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a05d: skipped 000328 1789266365.658504 mapwrite 0x009add thru 0x0168e6 (0xce0a bytes) 000329 1789266365.789404 trunc from 0x01a05d to 0x010fad (0x90af bytes) 000330 1789266365.824235 mapwrite 0x02a03c thru 0x02f917 (0x58dc bytes) 000331 1789266365.880674 MIRROR_RESYNC 10 mirrors 000332 1789266367.654214 trunc from 0x02f918 to 0x00db7c (0x21d9b bytes) 000333 1789266367.706804 trunc from 0x002773 to 0x00932f (0x6bbd bytes) 000334 1789266367.717652 mapread 0x000a33 thru 0x0084d9 (0x7aa7 bytes) 000335 1789266367.741974 trunc from 0x000df2 to 0x00d63c (0xc84b bytes) 000336 1789266367.749242 fallocate 0x026d56 thru 0x03088c (0x9b37 bytes) 000337 1789266367.754277 trunc from 0x005279 to 0x00db7c (0x8904 bytes) 000338 1789266367.758393 mapwrite 0x035289 thru 0x03d702 (0x847a bytes) 000339 1789266367.815425 write 0x021b7b thru 0x026a82 (0x4f08 bytes) 000340 1789266367.824833 mapwrite 0x01b253 thru 0x01fa62 (0x4810 bytes) 000341 1789266367.859843 punch from 0x01be60 to 0x01f44c (0x35ed bytes) 000342 1789266367.866197 fallocate 0x020154 thru 0x028e7d (0x8d2a bytes) 000343 1789266367.874647 trunc from 0x03d703 to 0x02284c (0x1aeb6 bytes) 000345 1789266367.894935 mapwrite 0x03ef4c thru 0x03ffff (0x10b4 bytes) 000346 1789266367.928211 mapwrite 0x03cfbe thru 0x03ffff (0x3042 bytes) 000347 1789266367.942629 mapread 0x0296d4 thru 0x02fd2f (0x665c bytes) 000348 1789266367.950925 fallocate 0x00fa9d thru 0x015b5e (0x60c2 bytes) 000349 1789266367.955780 mapwrite 0x01fb37 thru 0x022bb5 (0x307f bytes) 000350 1789266367.977584 trunc from 0x019f44 to 0x026b90 (0xcc4d bytes) 000351 1789266367.982993 write 0x014d62 thru 0x014e40 (0xdf bytes) 000352 1789266367.984316 write 0x020642 thru 0x0294d4 (0x8e93 bytes) 000353 1789266367.990587 fallocate 0x02d3fe thru 0x036c61 (0x9864 bytes) 000354 1789266367.995727 punch from 0x02dbaf to 0x030c1f (0x3071 bytes) 000355 1789266368.000706 read 0x02d239 thru 0x039c84 (0xca4c bytes) 000356 1789266368.013521 punch from 0x03dfc6 to 0x03ffff (0x203a bytes) 000357 1789266368.018558 trunc from 0x02a1ba to 0x03218c (0x7fd3 bytes) 000358 1789266368.023927 fallocate 0x031036 thru 0x03c385 (0xb350 bytes) 000359 1789266368.028888 trunc from 0x040000 to 0x0377e8 (0x8817 bytes) 000360 1789266368.047788 mapwrite 0x019006 thru 0x021028 (0x8023 bytes) 000361 1789266368.081820 fallocate 0x017d65 thru 0x022555 (0xa7f1 bytes) 000362 1789266368.087908 fallocate 0x008980 thru 0x012380 (0x9a01 bytes) 000363 1789266368.097145 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x377e9): Mirror 2: 0x97a9c0ef (stale) Mirror 4: 0x97a9c0ef (stale) Mirror 5: 0x97a9c0ef (stale) Mirror 6: 0x97a9c0ef (stale) Mirror 9: 0x97a9c0ef (stale) Mirror 10: 0x97a9c0ef (stale) Mirror 11: 0x97a9c0ef (stale) Mirror 13: 0x97a9c0ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x377e9): Mirror 2: 0x97a9c0ef (stale) Mirror 4: 0x97a9c0ef (stale) Mirror 5: 0x97a9c0ef (stale) Mirror 6: 0x97a9c0ef (stale) Mirror 9: 0x97a9c0ef (stale) Mirror 10: 0x97a9c0ef (stale) Mirror 11: 0x97a9c0ef (stale) Mirror 13: 0x97a9c0ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x377e9: skipped 000364 1789266368.699912 mapread 0x01bf86 thru 0x02198f (0x5a0a bytes) 000365 1789266368.727524 trunc from 0x027949 to 0x02bb64 (0x421c bytes) 000366 1789266368.737644 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x377e9): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x377e9): Mirror 2: 0x97a9c0ef (stale) Mirror 4: 0x97a9c0ef (stale) Mirror 5: 0x97a9c0ef (stale) Mirror 6: 0x97a9c0ef (stale) Mirror 9: 0x97a9c0ef (stale) Mirror 10: 0x97a9c0ef (stale) Mirror 11: 0x97a9c0ef (stale) Mirror 13: 0x97a9c0ef (stale) CRC-32 checksum value for chunk [0, 0x377e9): Mirror 1: 0x1007ea7b Mirror 14: 0x1007ea7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x377e9: skipped 000367 1789266369.891682 mapread 0x020589 thru 0x02b463 (0xaedb bytes) 000368 1789266369.947777 trunc from 0x00c3ca to 0x014402 (0x8039 bytes) 000369 1789266370.006705 punch from 0x010ac9 to 0x019fbb (0x94f3 bytes) 000370 1789266370.014246 mapread 0x01d7c5 thru 0x01e5f2 (0xe2e bytes) 000371 1789266370.029840 punch from 0x0267f2 to 0x0309cc (0xa1db bytes) 000372 1789266370.036752 mapread 0x01c260 thru 0x025a94 (0x9835 bytes) 000373 1789266370.038891 mapread 0x0189f3 thru 0x02466b (0xbc79 bytes) 000374 1789266370.041827 read 0x01a0bf thru 0x02458f (0xa4d1 bytes) 000375 1789266370.042725 write 0x00997f thru 0x016de9 (0xd46b bytes) 000376 1789266370.065693 read 0x00af90 thru 0x01985a (0xe8cb bytes) 000377 1789266370.066652 fallocate 0x025fec thru 0x0274cc (0x14e1 bytes) 000378 1789266370.090563 mapwrite 0x02f084 thru 0x034639 (0x55b6 bytes) 000379 1789266370.132918 mapwrite 0x0344f0 thru 0x03a54d (0x605e bytes) 000381 1789266370.182847 write 0x030c0d thru 0x03f8ba (0xecae bytes) 000382 1789266370.187646 mapread 0x01b8d8 thru 0x02860c (0xcd35 bytes) 000383 1789266370.188765 mapread 0x039f1d thru 0x03f8ba (0x599e bytes) 000385 1789266370.192043 mapread 0x039e9d thru 0x03f51f (0x5683 bytes) 000386 1789266370.195173 mapwrite 0x0006cc thru 0x000f72 (0x8a7 bytes) 000387 1789266370.219898 trunc from 0x03f8bb to 0x03c418 (0x34a2 bytes) 000388 1789266370.240925 write 0x0062e2 thru 0x006980 (0x69f bytes) 000389 1789266370.242831 trunc from 0x03c419 to 0x037331 (0x50e7 bytes) 000391 1789266370.267945 fallocate 0x01e96d thru 0x0250fe (0x6792 bytes) 000392 1789266370.273164 trunc from 0x02d12c to 0x0311db (0x40b0 bytes) 000393 1789266370.280558 punch from 0x0118bb to 0x0189cc (0x7112 bytes) 000394 1789266370.286483 mapwrite 0x0334ae thru 0x03eafb (0xb64e bytes) 000395 1789266370.346907 mapread 0x02e6be thru 0x02f823 (0x1166 bytes) 000396 1789266370.358278 trunc from 0x015ffc to 0x01a0e2 (0x40e7 bytes) 000397 1789266370.364177 trunc from 0x005bc8 to 0x00e297 (0x86d0 bytes) 000401 1789266370.370904 punch from 0x019436 to 0x01ed9e (0x5969 bytes) 000402 1789266370.376521 trunc from 0x000c5d to 0x0070e6 (0x648a bytes) 000404 1789266370.384482 fallocate 0x00d88b thru 0x014929 (0x709f bytes) 000405 1789266370.396642 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3eafc): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eafc): Mirror 2: 0xb31c649f (stale) Mirror 4: 0xb31c649f (stale) Mirror 5: 0xb31c649f (stale) Mirror 6: 0xb31c649f (stale) Mirror 9: 0xb31c649f (stale) Mirror 10: 0xb31c649f (stale) Mirror 11: 0xb31c649f (stale) Mirror 13: 0xb31c649f (stale) Mirror 14: 0xa55d4a6a (stale) CRC-32 checksum value for chunk [0, 0x3eafc): Mirror 1: 0xa192b9c8 Mirror 15: 0xa192b9c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eafc: skipped 000406 1789266371.367818 trunc from 0x0168d8 to 0x01fe58 (0x9581 bytes) 000407 1789266371.425580 read 0x01ae0c thru 0x025d1e (0xaf13 bytes) 000408 1789266371.434304 read 0x025d69 thru 0x02a49b (0x4733 bytes) 000409 1789266371.445041 trunc from 0x03eafc to 0x0122ee (0x2c80d bytes) 000410 1789266371.486076 mapread 0x00ea8c thru 0x0122ee (0x3863 bytes) 000411 1789266371.491401 MIRROR_RESYNC 11 mirrors 000412 1789266373.853190 punch from 0x00f7d4 to 0x0122ee (0x2b1b bytes) 000413 1789266373.913510 punch from 0x00446d to 0x00c978 (0x850c bytes) 000414 1789266373.924604 read 0x0048b4 thru 0x00690f (0x205c bytes) 000415 1789266373.931097 punch from 0x004b2f to 0x00fd53 (0xb225 bytes) 000416 1789266373.939360 punch from 0x00f4f7 to 0x0122ee (0x2df8 bytes) 000417 1789266373.946764 trunc from 0x0122ef to 0x0149d6 (0x26e8 bytes) 000418 1789266373.990548 punch from 0x0096e5 to 0x012493 (0x8daf bytes) 000419 1789266373.995377 punch from 0x0132bf to 0x0149d6 (0x1718 bytes) 000420 1789266373.999799 trunc from 0x0149d7 to 0x01c716 (0x7d40 bytes) 000421 1789266374.016197 trunc from 0x01c717 to 0x021872 (0x515c bytes) 000422 1789266374.030260 read 0x003fca thru 0x00d28f (0x92c6 bytes) 000423 1789266374.040785 mapread 0x0053aa thru 0x00954b (0x41a2 bytes) 000424 1789266374.041837 write 0x022e43 thru 0x02e988 (0xbb46 bytes) 000425 1789266374.043873 read 0x01e18d thru 0x02b232 (0xd0a6 bytes) 000426 1789266374.045207 mapread 0x007252 thru 0x010bf8 (0x99a7 bytes) 000427 1789266374.046563 fallocate 0x037a99 thru 0x03fa8b (0x7ff3 bytes) 000428 1789266374.051182 read 0x000390 thru 0x000e8e (0xaff bytes) 000429 1789266374.051787 write 0x02e6af thru 0x03b9a3 (0xd2f5 bytes) 000430 1789266374.057295 mapread 0x0236dd thru 0x033676 (0xff9a bytes) 000431 1789266374.058516 read 0x0284c9 thru 0x031dc0 (0x98f8 bytes) 000433 1789266374.060144 mapwrite 0x01933d thru 0x01f946 (0x660a bytes) 000434 1789266374.084678 mapwrite 0x02d7e5 thru 0x03502e (0x784a bytes) 000436 1789266374.109080 read 0x0072b0 thru 0x01026c (0x8fbd bytes) 000437 1789266374.110186 mapread 0x0361a3 thru 0x03fa8b (0x98e9 bytes) 000438 1789266374.118101 mapwrite 0x017d4e thru 0x018130 (0x3e3 bytes) 000439 1789266374.130427 mapread 0x03e66a thru 0x03fa8b (0x1422 bytes) 000440 1789266374.131765 mapwrite 0x01ba58 thru 0x02b785 (0xfd2e bytes) 000441 1789266374.152664 punch from 0x01fa98 to 0x02f175 (0xf6de bytes) 000442 1789266374.159057 mapwrite 0x036488 thru 0x03ebd8 (0x8751 bytes) 000443 1789266374.179383 mapwrite 0x01a846 thru 0x022b6f (0x832a bytes) 000444 1789266374.242616 write 0x0326e2 thru 0x035e44 (0x3763 bytes) 000446 1789266374.244175 fallocate 0x018747 thru 0x018c34 (0x4ee bytes) 000447 1789266374.254543 MIRROR_RESYNC 11 mirrors 000448 1789266376.829637 punch from 0x0048fb to 0x006f4a (0x2650 bytes) 000449 1789266376.878169 write 0x01c556 thru 0x01f522 (0x2fcd bytes) 000450 1789266376.893707 mapwrite 0x0336e9 thru 0x03ffff (0xc917 bytes) 000451 1789266376.967681 punch from 0x03294e to 0x03eb89 (0xc23c bytes) 000452 1789266376.972812 fallocate 0x02aba3 thru 0x03a4be (0xf91c bytes) 000453 1789266376.977472 trunc from 0x040000 to 0x014558 (0x2baa7 bytes) 000455 1789266376.995424 fallocate 0x004638 thru 0x00c445 (0x7e0e bytes) 000456 1789266377.001450 mapread 0x008c98 thru 0x0102a1 (0x760a bytes) 000457 1789266377.017692 MIRROR_RESYNC 11 mirrors 000458 1789266378.438166 trunc from 0x014559 to 0x01cc8e (0x8736 bytes) 000459 1789266378.495018 fallocate 0x00ee9e thru 0x00f109 (0x26c bytes) 000461 1789266378.500920 fallocate 0x00b0d8 thru 0x00c8e3 (0x180c bytes) 000462 1789266378.507191 read 0x002ac0 thru 0x0101ae (0xd6ef bytes) 000463 1789266378.526441 read 0x014176 thru 0x01aa28 (0x68b3 bytes) 000464 1789266378.541795 mapwrite 0x01d1fa thru 0x029f48 (0xcd4f bytes) 000465 1789266378.633325 mapread 0x01a40f thru 0x01b2a1 (0xe93 bytes) 000466 1789266378.635665 mapread 0x026def thru 0x029f48 (0x315a bytes) 000468 1789266378.637899 read 0x012de7 thru 0x01397a (0xb94 bytes) 000469 1789266378.638732 read 0x0186c1 thru 0x0282a0 (0xfbe0 bytes) 000470 1789266378.639514 punch from 0x02570c to 0x0285a8 (0x2e9d bytes) 000472 1789266378.646991 read 0x002b7c thru 0x009846 (0x6ccb bytes) 000473 1789266378.648157 trunc from 0x029f49 to 0x01801d (0x11f2b bytes) 000474 1789266378.662173 write 0x024890 thru 0x02cdef (0x8560 bytes) 000475 1789266378.674773 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2cdf0): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cdf0): Mirror 2: 0xf87ab33a (stale) Mirror 4: 0xf87ab33a (stale) Mirror 5: 0xf87ab33a (stale) Mirror 6: 0xf87ab33a (stale) Mirror 9: 0xf87ab33a (stale) Mirror 10: 0xf87ab33a (stale) Mirror 11: 0xf87ab33a (stale) Mirror 13: 0xf87ab33a (stale) Mirror 14: 0xf87ab33a (stale) Mirror 15: 0xf87ab33a (stale) CRC-32 checksum value for chunk [0, 0x2cdf0): Mirror 1: 0x5ef3f713 Mirror 16: 0x5ef3f713 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cdf0: skipped 000476 1789266379.748688 read 0x02a7b3 thru 0x02cdef (0x263d bytes) 000477 1789266379.766998 write 0x00e30d thru 0x0143a0 (0x6094 bytes) 000478 1789266379.814075 mapwrite 0x016b66 thru 0x01e1fa (0x7695 bytes) 000479 1789266379.878513 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2cdf0): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cdf0): Mirror 2: 0xf87ab33a (stale) Mirror 4: 0xf87ab33a (stale) Mirror 5: 0xf87ab33a (stale) Mirror 6: 0xf87ab33a (stale) Mirror 9: 0xf87ab33a (stale) Mirror 10: 0xf87ab33a (stale) Mirror 11: 0xf87ab33a (stale) Mirror 13: 0xf87ab33a (stale) Mirror 14: 0xf87ab33a (stale) Mirror 15: 0xf87ab33a (stale) Mirror 16: 0x5ef3f713 (stale) CRC-32 checksum value for chunk [0, 0x2cdf0): Mirror 1: 0xba6e5498 Mirror 17: 0xba6e5498 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cdf0: skipped 000480 1789266381.004875 fallocate 0x0167b7 thru 0x016918 (0x162 bytes) 000481 1789266381.051262 fallocate 0x027e8e thru 0x02c808 (0x497b bytes) 000482 1789266381.056272 punch from 0x00f5ed to 0x010e30 (0x1844 bytes) 000483 1789266381.062020 trunc from 0x008676 to 0x018003 (0xf98e bytes) 000484 1789266381.067380 mapwrite 0x00a758 thru 0x017646 (0xceef bytes) 000485 1789266381.122266 punch from 0x01f0ad to 0x02c4f0 (0xd444 bytes) 000486 1789266381.127244 mapread 0x017007 thru 0x0185e0 (0x15da bytes) 000487 1789266381.128616 read 0x005cf8 thru 0x00603c (0x345 bytes) 000488 1789266381.129278 trunc from 0x020a0c to 0x02cdef (0xc3e4 bytes) 000489 1789266381.134303 mapwrite 0x000aee thru 0x008f41 (0x8454 bytes) 000490 1789266381.159200 read 0x026224 thru 0x02cdef (0x6bcc bytes) 000491 1789266381.175178 write 0x029274 thru 0x02c763 (0x34f0 bytes) 000492 1789266381.178987 fallocate 0x018a42 thru 0x024447 (0xba06 bytes) 000493 1789266381.185069 mapwrite 0x004f10 thru 0x013063 (0xe154 bytes) 000494 1789266381.210570 mapread 0x02355d thru 0x029c7f (0x6723 bytes) 000496 1789266381.218515 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cdf0): Mirror 2: 0xf87ab33a (stale) Mirror 4: 0xf87ab33a (stale) Mirror 5: 0xf87ab33a (stale) Mirror 6: 0xf87ab33a (stale) Mirror 9: 0xf87ab33a (stale) Mirror 10: 0xf87ab33a (stale) Mirror 11: 0xf87ab33a (stale) Mirror 13: 0xf87ab33a (stale) Mirror 14: 0xf87ab33a (stale) Mirror 16: 0x5ef3f713 (stale) Mirror 17: 0xba6e5498 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cdf0): Mirror 2: 0xf87ab33a (stale) Mirror 4: 0xf87ab33a (stale) Mirror 5: 0xf87ab33a (stale) Mirror 6: 0xf87ab33a (stale) Mirror 9: 0xf87ab33a (stale) Mirror 10: 0xf87ab33a (stale) Mirror 11: 0xf87ab33a (stale) Mirror 13: 0xf87ab33a (stale) Mirror 14: 0xf87ab33a (stale) Mirror 16: 0x5ef3f713 (stale) Mirror 17: 0xba6e5498 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cdf0: skipped 000497 1789266381.851907 trunc from 0x02cdf0 to 0x03d5fa (0x1080b bytes) 000498 1789266381.871550 fallocate 0x006ecc thru 0x010fce (0xa103 bytes) 000499 1789266381.880734 write 0x0038d0 thru 0x00737f (0x3ab0 bytes) 000501 1789266381.904688 read 0x02aca0 thru 0x02aeec (0x24d bytes) 000502 1789266381.913612 mapread 0x025947 thru 0x02cd28 (0x73e2 bytes) 000503 1789266381.942559 trunc from 0x03d5fb to 0x02076c (0x1ce8e bytes) 000504 1789266381.963886 trunc from 0x02076d to 0x01aa0b (0x5d61 bytes) 000505 1789266381.984554 punch from 0x0049e0 to 0x005c10 (0x1231 bytes) 000506 1789266382.000671 punch from 0x0196ea to 0x01aa0b (0x1322 bytes) 000507 1789266382.006714 mapread 0x000a35 thru 0x00833e (0x790a bytes) 000509 1789266382.021665 fallocate 0x02806f thru 0x029bea (0x1b7c bytes) 000510 1789266382.032016 trunc from 0x000290 to 0x008802 (0x8573 bytes) 000511 1789266382.046523 mapread 0x017995 thru 0x01aa0b (0x3077 bytes) 000513 1789266382.068079 trunc from 0x01aa0c to 0x036b04 (0x1c0f9 bytes) 000514 1789266382.087324 punch from 0x005b7f to 0x013679 (0xdafb bytes) 000515 1789266382.095596 trunc from 0x036b05 to 0x000e6e (0x35c96 bytes) 000516 1789266382.116920 mapwrite 0x03d8ae thru 0x03ffff (0x2752 bytes) 000517 1789266382.163746 fallocate 0x030375 thru 0x03ad8c (0xaa18 bytes) 000518 1789266382.173485 mapwrite 0x00f901 thru 0x018bc3 (0x92c3 bytes) 000519 1789266382.227766 trunc from 0x034b02 to 0x036127 (0x1626 bytes) 000520 1789266382.235365 mapwrite 0x00acbf thru 0x01a708 (0xfa4a bytes) 000521 1789266382.261634 mapread 0x01ee74 thru 0x0205e2 (0x176f bytes) 000522 1789266382.262588 punch from 0x00dd3d to 0x01611f (0x83e3 bytes) 000523 1789266382.286697 trunc from 0x040000 to 0x021b80 (0x1e47f bytes) 000524 1789266382.311540 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x21b81): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21b81): Mirror 2: 0x1d315bab (stale) Mirror 4: 0x1d315bab (stale) Mirror 5: 0x1d315bab (stale) Mirror 6: 0x1d315bab (stale) Mirror 9: 0x1d315bab (stale) Mirror 10: 0x1d315bab (stale) Mirror 11: 0x1d315bab (stale) Mirror 13: 0x1d315bab (stale) Mirror 14: 0x1d315bab (stale) Mirror 16: 0x1d315bab (stale) Mirror 17: 0x8259d179 (stale) CRC-32 checksum value for chunk [0, 0x21b81): Mirror 1: 0xe43eff7e Mirror 18: 0xe43eff7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21b81: skipped 000525 1789266383.465084 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21b81): Mirror 2: 0x1d315bab (stale) Mirror 4: 0x1d315bab (stale) Mirror 5: 0x1d315bab (stale) Mirror 6: 0x1d315bab (stale) Mirror 9: 0x1d315bab (stale) Mirror 11: 0x1d315bab (stale) Mirror 13: 0x1d315bab (stale) Mirror 14: 0x1d315bab (stale) Mirror 16: 0x1d315bab (stale) Mirror 17: 0x8259d179 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21b81): Mirror 2: 0x1d315bab (stale) Mirror 4: 0x1d315bab (stale) Mirror 5: 0x1d315bab (stale) Mirror 6: 0x1d315bab (stale) Mirror 9: 0x1d315bab (stale) Mirror 11: 0x1d315bab (stale) Mirror 13: 0x1d315bab (stale) Mirror 14: 0x1d315bab (stale) Mirror 16: 0x1d315bab (stale) Mirror 17: 0x8259d179 (stale) CRC-32 checksum value for chunk [0, 0x21b81): Mirror 1: 0xe43eff7e Mirror 18: 0xe43eff7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21b81: skipped 000526 1789266384.495184 mapwrite 0x01960c thru 0x028995 (0xf38a bytes) 000527 1789266384.654875 read 0x0144d8 thru 0x01f79f (0xb2c8 bytes) 000528 1789266384.655939 fallocate 0x03682b thru 0x03ffff (0x97d5 bytes) 000529 1789266384.664141 read 0x00b86c thru 0x018465 (0xcbfa bytes) 000530 1789266384.678236 trunc from 0x040000 to 0x0371c8 (0x8e37 bytes) 000531 1789266384.705226 write 0x02859f thru 0x032d76 (0xa7d8 bytes) 000532 1789266384.707603 mapwrite 0x01ce34 thru 0x02ba58 (0xec25 bytes) 000533 1789266384.747540 write 0x03dc0a thru 0x03ffff (0x23f6 bytes) 000534 1789266384.748659 mapwrite 0x0205b0 thru 0x0294dc (0x8f2d bytes) 000535 1789266384.809680 trunc from 0x03c975 to 0x03ffff (0x368b bytes) 000536 1789266384.815456 mapwrite 0x02292e thru 0x023541 (0xc14 bytes) 000537 1789266384.830612 read 0x028b5a thru 0x035734 (0xcbdb bytes) 000538 1789266384.831306 write 0x00e8f6 thru 0x017c3d (0x9348 bytes) 000539 1789266384.836960 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xac0f45c5 (stale) Mirror 4: 0xac0f45c5 (stale) Mirror 5: 0xac0f45c5 (stale) Mirror 6: 0xac0f45c5 (stale) Mirror 9: 0xac0f45c5 (stale) Mirror 11: 0xac0f45c5 (stale) Mirror 13: 0xac0f45c5 (stale) Mirror 14: 0xac0f45c5 (stale) Mirror 16: 0xc854c86e (stale) Mirror 17: 0x58a809c6 (stale) Mirror 18: 0xd702f88c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x279d009c Mirror 19: 0x279d009c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000540 1789266386.080087 mapwrite 0x034b7b thru 0x03fe8f (0xb315 bytes) 000542 1789266386.282842 trunc from 0x02b2ae to 0x02ce42 (0x1b95 bytes) 000543 1789266386.312245 MIRROR_RESYNC 13 mirrors 000544 1789266389.272319 MIRROR_EXTEND to 14 mirrors 000545 1789266390.269455 read 0x03342f thru 0x03cc70 (0x9842 bytes) 000546 1789266390.292732 fallocate 0x024ad9 thru 0x02b061 (0x6589 bytes) 000547 1789266390.382561 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x32bdbe51 (stale) Mirror 4: 0x32bdbe51 (stale) Mirror 5: 0x32bdbe51 (stale) Mirror 6: 0x32bdbe51 (stale) Mirror 9: 0x32bdbe51 (stale) Mirror 11: 0x32bdbe51 (stale) Mirror 13: 0x32bdbe51 (stale) Mirror 14: 0x32bdbe51 (stale) Mirror 16: 0x32bdbe51 (stale) Mirror 17: 0x32bdbe51 (stale) Mirror 18: 0x32bdbe51 (stale) Mirror 19: 0x32bdbe51 (stale) Mirror 20: 0x32bdbe51 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x32bdbe51 Mirror 21: 0x32bdbe51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000550 1789266391.767860 mapread 0x021bdd thru 0x02ab2c (0x8f50 bytes) 000551 1789266391.828422 fallocate 0x019ef0 thru 0x01c388 (0x2499 bytes) 000553 1789266391.878142 trunc from 0x01cba0 to 0x0241df (0x7640 bytes) 000554 1789266391.883826 read 0x022844 thru 0x03225a (0xfa17 bytes) 000555 1789266391.891473 read 0x019b22 thru 0x025eea (0xc3c9 bytes) 000556 1789266391.902068 write 0x022bce thru 0x0316ef (0xeb22 bytes) 000557 1789266391.908671 mapread 0x01b86c thru 0x027231 (0xb9c6 bytes) 000559 1789266391.911197 punch from 0x01e1f3 to 0x029b9e (0xb9ac bytes) 000560 1789266391.927156 punch from 0x02c090 to 0x03aa2e (0xe99f bytes) 000561 1789266391.933056 trunc from 0x002d2b to 0x00fbba (0xce90 bytes) 000562 1789266391.938453 trunc from 0x040000 to 0x02edee (0x11211 bytes) 000563 1789266391.955055 punch from 0x02584e to 0x02edee (0x95a1 bytes) 000564 1789266391.960150 mapwrite 0x027b04 thru 0x036e73 (0xf370 bytes) 000565 1789266392.078846 punch from 0x033b93 to 0x036e73 (0x32e1 bytes) 000566 1789266392.084121 write 0x038a98 thru 0x03b456 (0x29bf bytes) 000567 1789266392.097968 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3b457): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b457): Mirror 2: 0x1bdaf2db (stale) Mirror 4: 0x1bdaf2db (stale) Mirror 5: 0x1bdaf2db (stale) Mirror 6: 0x1bdaf2db (stale) Mirror 9: 0x1bdaf2db (stale) Mirror 11: 0x1bdaf2db (stale) Mirror 13: 0x1bdaf2db (stale) Mirror 14: 0x1bdaf2db (stale) Mirror 16: 0x1bdaf2db (stale) Mirror 17: 0x1bdaf2db (stale) Mirror 18: 0x1bdaf2db (stale) Mirror 19: 0x1bdaf2db (stale) Mirror 20: 0x1bdaf2db (stale) Mirror 21: 0x1bdaf2db (stale) CRC-32 checksum value for chunk [0, 0x3b457): Mirror 1: 0x5505522f Mirror 22: 0x5505522f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b457: skipped 000568 1789266393.272315 read 0x00d23a thru 0x00fd76 (0x2b3d bytes) 000570 1789266393.298786 trunc from 0x020235 to 0x020a84 (0x850 bytes) 000571 1789266393.368308 fallocate 0x035126 thru 0x03b48b (0x6366 bytes) 000572 1789266393.379700 write 0x00e7b0 thru 0x010a0e (0x225f bytes) 000574 1789266393.402504 punch from 0x0359dd to 0x03b48b (0x5aaf bytes) 000575 1789266393.416925 mapread 0x0117d4 thru 0x01eedd (0xd70a bytes) 000576 1789266393.490514 write 0x01d485 thru 0x0242ad (0x6e29 bytes) 000577 1789266393.492063 write 0x016a7b thru 0x01fb2e (0x90b4 bytes) 000579 1789266393.514009 MIRROR_RESYNC 16 mirrors 000580 1789266397.726064 trunc from 0x03b48c to 0x01fc82 (0x1b809 bytes) 000584 1789266397.778139 trunc from 0x00653d to 0x0078e2 (0x13a6 bytes) 000585 1789266397.783894 fallocate 0x01fdfe thru 0x02a01e (0xa221 bytes) 000586 1789266397.789412 punch from 0x01d8cf to 0x01fc82 (0x23b4 bytes) 000587 1789266397.795698 mapwrite 0x018f9d thru 0x01c8e0 (0x3944 bytes) 000588 1789266397.830203 punch from 0x01248c to 0x01fc82 (0xd7f7 bytes) 000589 1789266397.838032 mapwrite 0x023c64 thru 0x031005 (0xd3a2 bytes) 000590 1789266397.926997 fallocate 0x0232d3 thru 0x023cce (0x9fc bytes) 000591 1789266397.935096 fallocate 0x00a04b thru 0x00f23f (0x51f5 bytes) 000592 1789266397.941967 mapread 0x00eaad thru 0x0140b9 (0x560d bytes) 000593 1789266397.963673 mapread 0x005d60 thru 0x00bf29 (0x61ca bytes) 000594 1789266397.981195 fallocate 0x03d116 thru 0x03ffff (0x2eea bytes) 000595 1789266397.987317 fallocate 0x02c402 thru 0x02f0ef (0x2cee bytes) 000596 1789266397.991839 mapread 0x02a6d9 thru 0x02c48b (0x1db3 bytes) 000597 1789266397.993032 fallocate 0x02a6a8 thru 0x02ec86 (0x45df bytes) 000600 1789266398.006692 MIRROR_RESYNC 16 mirrors 000601 1789266401.668145 fallocate 0x01d47d thru 0x01e185 (0xd09 bytes) 000602 1789266401.727020 read 0x00370d thru 0x0101ca (0xcabe bytes) 000603 1789266401.741040 read 0x009f67 thru 0x013791 (0x982b bytes) 000605 1789266401.764275 mapwrite 0x034fa4 thru 0x03fa00 (0xaa5d bytes) 000606 1789266401.841422 punch from 0x029749 to 0x030cd2 (0x758a bytes) 000607 1789266401.847906 mapwrite 0x02cd8f thru 0x02e3ec (0x165e bytes) 000609 1789266401.907868 punch from 0x011411 to 0x01ffab (0xeb9b bytes) 000610 1789266401.923918 MIRROR_RESYNC 16 mirrors 000612 1789266407.734182 mapwrite 0x02060b thru 0x028129 (0x7b1f bytes) 000613 1789266407.837926 mapread 0x00bb06 thru 0x01a2bc (0xe7b7 bytes) 000614 1789266407.865025 fallocate 0x016aee thru 0x022b8f (0xc0a2 bytes) 000615 1789266407.872904 trunc from 0x03fa01 to 0x038eff (0x6b01 bytes) 000616 1789266407.897747 punch from 0x01223e to 0x01c95e (0xa721 bytes) 000617 1789266407.905532 read 0x01d088 thru 0x0236d0 (0x6649 bytes) 000618 1789266407.906980 mapread 0x02d8de thru 0x03612c (0x884f bytes) 000619 1789266407.922491 fallocate 0x00fe0c thru 0x017c00 (0x7df5 bytes) 000620 1789266407.927546 punch from 0x00b30b to 0x00f696 (0x438c bytes) 000621 1789266407.932978 mapwrite 0x03ec7c thru 0x03ffff (0x1384 bytes) 000622 1789266408.002327 trunc from 0x040000 to 0x00969e (0x36961 bytes) 000624 1789266408.022748 mapread 0x001ee5 thru 0x00683f (0x495b bytes) 000625 1789266408.040846 write 0x03ec07 thru 0x03ffff (0x13f9 bytes) 000626 1789266408.043215 read 0x0275e1 thru 0x028e83 (0x18a3 bytes) 000627 1789266408.052052 read 0x002894 thru 0x00fa85 (0xd1f2 bytes) 000628 1789266408.053140 write 0x000b5f thru 0x009b88 (0x902a bytes) 000629 1789266408.061001 read 0x02e4f3 thru 0x035327 (0x6e35 bytes) 000630 1789266408.061718 fallocate 0x00bef8 thru 0x00c51e (0x627 bytes) 000631 1789266408.070562 punch from 0x0296c3 to 0x02fbd4 (0x6512 bytes) 000632 1789266408.076856 trunc from 0x020a4a to 0x0220d3 (0x168a bytes) 000633 1789266408.081028 fallocate 0x025098 thru 0x0258ba (0x823 bytes) 000634 1789266408.086045 mapread 0x013d76 thru 0x01719d (0x3428 bytes) 000635 1789266408.087180 punch from 0x0199fa to 0x023141 (0x9748 bytes) 000636 1789266408.092977 trunc from 0x040000 to 0x00d415 (0x32bea bytes) 000637 1789266408.106020 trunc from 0x00d416 to 0x02b222 (0x1de0d bytes) 000638 1789266408.123816 mapwrite 0x0382e4 thru 0x03abea (0x2907 bytes) 000639 1789266408.171289 write 0x03f88a thru 0x03ffff (0x776 bytes) 000641 1789266408.172873 trunc from 0x040000 to 0x03935c (0x6ca3 bytes) 000642 1789266408.197708 mapwrite 0x00bfa2 thru 0x015659 (0x96b8 bytes) 000643 1789266408.218788 mapread 0x01ab93 thru 0x01d325 (0x2793 bytes) 000644 1789266408.224055 MIRROR_SPLIT mirror 12 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3935d): Mirror 2: 0x98231cf0 (stale) Mirror 4: 0x98231cf0 (stale) Mirror 5: 0x98231cf0 (stale) Mirror 6: 0x98231cf0 (stale) Mirror 9: 0x98231cf0 (stale) Mirror 11: 0x98231cf0 (stale) Mirror 13: 0x98231cf0 (stale) Mirror 14: 0x98231cf0 (stale) Mirror 16: 0x98231cf0 (stale) Mirror 17: 0x98231cf0 (stale) Mirror 18: 0x98231cf0 (stale) Mirror 20: 0x98231cf0 (stale) Mirror 21: 0x98231cf0 (stale) Mirror 22: 0x98231cf0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3935d): Mirror 2: 0x98231cf0 (stale) Mirror 4: 0x98231cf0 (stale) Mirror 5: 0x98231cf0 (stale) Mirror 6: 0x98231cf0 (stale) Mirror 9: 0x98231cf0 (stale) Mirror 11: 0x98231cf0 (stale) Mirror 13: 0x98231cf0 (stale) Mirror 14: 0x98231cf0 (stale) Mirror 16: 0x98231cf0 (stale) Mirror 17: 0x98231cf0 (stale) Mirror 18: 0x98231cf0 (stale) Mirror 20: 0x98231cf0 (stale) Mirror 21: 0x98231cf0 (stale) Mirror 22: 0x98231cf0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3935d: skipped 000646 1789266408.894184 mapwrite 0x02b062 thru 0x035555 (0xa4f4 bytes) 000647 1789266408.955592 read 0x007a58 thru 0x009033 (0x15dc bytes) 000648 1789266408.960554 fallocate 0x00b7fa thru 0x0100eb (0x48f2 bytes) 000649 1789266408.965192 write 0x01510f thru 0x01ead5 (0x99c7 bytes) 000650 1789266408.977721 mapread 0x027498 thru 0x02fc9f (0x8808 bytes) 000651 1789266408.978758 mapread 0x01ce32 thru 0x027f88 (0xb157 bytes) 000653 1789266408.987051 fallocate 0x025243 thru 0x0287fb (0x35b9 bytes) 000654 1789266408.991824 trunc from 0x03935d to 0x02e98c (0xa9d0 bytes) 000655 1789266409.006463 mapread 0x010c08 thru 0x01c734 (0xbb2d bytes) 000656 1789266409.017518 read 0x01b72d thru 0x02754b (0xbe1f bytes) 000657 1789266409.021799 MIRROR_RESYNC 15 mirrors 000658 1789266412.009412 mapwrite 0x01512e thru 0x016e95 (0x1d68 bytes) 000659 1789266412.107723 fallocate 0x00fed5 thru 0x0147d4 (0x4900 bytes) 000660 1789266412.114617 mapwrite 0x0290be thru 0x02d92c (0x486f bytes) 000661 1789266412.140712 trunc from 0x02e98d to 0x02e54e (0x43e bytes) 000662 1789266412.166713 fallocate 0x03430d thru 0x03ffff (0xbcf3 bytes) 000665 1789266412.173175 mapwrite 0x008b4f thru 0x016d2e (0xe1e0 bytes) 000666 1789266412.205689 write 0x03904f thru 0x03bcfa (0x2cac bytes) 000667 1789266412.216901 fallocate 0x0294fa thru 0x02a721 (0x1228 bytes) 000668 1789266412.222836 mapwrite 0x03845f thru 0x03ffff (0x7ba1 bytes) 000669 1789266412.254797 fallocate 0x01fe6c thru 0x0230f7 (0x328c bytes) 000670 1789266412.260748 trunc from 0x0051ab to 0x014c30 (0xfa86 bytes) 000671 1789266412.268681 punch from 0x00831d to 0x014323 (0xc007 bytes) 000672 1789266412.281644 write 0x018cd7 thru 0x01c54e (0x3878 bytes) 000673 1789266412.291734 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3e6120c3 (stale) Mirror 4: 0x3e6120c3 (stale) Mirror 5: 0x3e6120c3 (stale) Mirror 6: 0x3e6120c3 (stale) Mirror 9: 0x3e6120c3 (stale) Mirror 11: 0x3e6120c3 (stale) Mirror 13: 0x3e6120c3 (stale) Mirror 14: 0x3e6120c3 (stale) Mirror 16: 0x3e6120c3 (stale) Mirror 17: 0x3e6120c3 (stale) Mirror 18: 0x3e6120c3 (stale) Mirror 20: 0x3e6120c3 (stale) Mirror 21: 0x3e6120c3 (stale) Mirror 22: 0x3e6120c3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x67cb66ed Mirror 23: 0x67cb66ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000674 1789266413.472109 trunc from 0x040000 to 0x030a25 (0xf5da bytes) 000675 1789266413.578639 read 0x009c75 thru 0x017a1a (0xdda6 bytes) 000676 1789266413.600392 trunc from 0x01da06 to 0x020629 (0x2c24 bytes) 000677 1789266413.607056 trunc from 0x030a26 to 0x01188b (0x1f19a bytes) 000678 1789266413.649048 MIRROR_SPLIT mirror 5 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1188c): Mirror 2: 0x751d5a1e (stale) Mirror 4: 0x751d5a1e (stale) Mirror 5: 0x751d5a1e (stale) Mirror 6: 0x751d5a1e (stale) Mirror 11: 0x751d5a1e (stale) Mirror 13: 0x751d5a1e (stale) Mirror 14: 0x751d5a1e (stale) Mirror 16: 0x751d5a1e (stale) Mirror 17: 0x751d5a1e (stale) Mirror 18: 0x751d5a1e (stale) Mirror 20: 0x751d5a1e (stale) Mirror 21: 0x751d5a1e (stale) Mirror 22: 0x751d5a1e (stale) Mirror 23: 0x64b28f43 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1188c): Mirror 2: 0x751d5a1e (stale) Mirror 4: 0x751d5a1e (stale) Mirror 5: 0x751d5a1e (stale) Mirror 6: 0x751d5a1e (stale) Mirror 11: 0x751d5a1e (stale) Mirror 13: 0x751d5a1e (stale) Mirror 14: 0x751d5a1e (stale) Mirror 16: 0x751d5a1e (stale) Mirror 17: 0x751d5a1e (stale) Mirror 18: 0x751d5a1e (stale) Mirror 20: 0x751d5a1e (stale) Mirror 21: 0x751d5a1e (stale) Mirror 22: 0x751d5a1e (stale) Mirror 23: 0x64b28f43 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1188c: skipped 000679 1789266414.544698 write 0x02d1fc thru 0x03a6cd (0xd4d2 bytes) 000680 1789266414.553442 trunc from 0x020ea9 to 0x021048 (0x1a0 bytes) 000681 1789266414.558918 punch from 0x00cfc1 to 0x0198d3 (0xc913 bytes) 000683 1789266414.565657 read 0x01712e thru 0x024c56 (0xdb29 bytes) 000686 1789266414.583798 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3a6ce): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a6ce): Mirror 2: 0xa90984f1 (stale) Mirror 4: 0xa90984f1 (stale) Mirror 5: 0xa90984f1 (stale) Mirror 6: 0xa90984f1 (stale) Mirror 11: 0xa90984f1 (stale) Mirror 13: 0xa90984f1 (stale) Mirror 14: 0xa90984f1 (stale) Mirror 16: 0xa90984f1 (stale) Mirror 17: 0xa90984f1 (stale) Mirror 18: 0xa90984f1 (stale) Mirror 20: 0xa90984f1 (stale) Mirror 21: 0xa90984f1 (stale) Mirror 22: 0xa90984f1 (stale) Mirror 23: 0x9b1a1c24 (stale) CRC-32 checksum value for chunk [0, 0x3a6ce): Mirror 1: 0xd6274b21 Mirror 24: 0xd6274b21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a6ce: skipped 000687 1789266415.738180 punch from 0x02fb3a to 0x035267 (0x572e bytes) 000688 1789266415.784578 mapread 0x034063 thru 0x03a6cd (0x666b bytes) 000689 1789266415.814393 punch from 0x003e78 to 0x010ea1 (0xd02a bytes) 000690 1789266415.826870 trunc from 0x0389b4 to 0x03a6cd (0x1d1a bytes) 000691 1789266415.832963 mapwrite 0x02b005 thru 0x02e5f4 (0x35f0 bytes) 000692 1789266415.871786 write 0x00cf99 thru 0x01707d (0xa0e5 bytes) 000693 1789266415.889658 trunc from 0x01abb8 to 0x023ec6 (0x930f bytes) 000694 1789266415.894318 punch from 0x02b98d to 0x03011c (0x4790 bytes) 000695 1789266415.903931 mapwrite 0x02a275 thru 0x039884 (0xf610 bytes) 000696 1789266415.949026 trunc from 0x00db72 to 0x0122f1 (0x4780 bytes) 000697 1789266415.959291 mapwrite 0x01d3ea thru 0x01f565 (0x217c bytes) 000698 1789266415.991585 MIRROR_RESYNC 16 mirrors 000699 1789266420.961481 read 0x0277f1 thru 0x02977d (0x1f8d bytes) 000700 1789266420.967821 trunc from 0x03a6ce to 0x03080b (0x9ec2 bytes) 000701 1789266421.024540 mapread 0x012830 thru 0x01ed5f (0xc530 bytes) 000702 1789266421.053889 write 0x02d91d thru 0x032868 (0x4f4c bytes) 000703 1789266421.060553 fallocate 0x00f768 thru 0x01a575 (0xae0e bytes) 000705 1789266421.066119 read 0x02ff51 thru 0x032868 (0x2918 bytes) 000706 1789266421.066777 punch from 0x010391 to 0x0172b1 (0x6f21 bytes) 000707 1789266421.073559 mapwrite 0x03050c thru 0x03e96a (0xe45f bytes) 000709 1789266421.144313 read 0x02f8de thru 0x031dda (0x24fd bytes) 000710 1789266421.145843 punch from 0x00eb46 to 0x011395 (0x2850 bytes) 000712 1789266421.154633 write 0x023e63 thru 0x03261a (0xe7b8 bytes) 000713 1789266421.158234 write 0x01a943 thru 0x027dcd (0xd48b bytes) 000714 1789266421.160840 fallocate 0x01a71a thru 0x027425 (0xcd0c bytes) 000715 1789266421.166565 mapread 0x0365d0 thru 0x039247 (0x2c78 bytes) 000716 1789266421.168292 trunc from 0x03e96b to 0x021a9f (0x1cecb bytes) 000717 1789266421.188738 mapwrite 0x03bc50 thru 0x03f6ac (0x3a5d bytes) 000718 1789266421.229858 trunc from 0x009e59 to 0x00be62 (0x200a bytes) 000720 1789266421.241689 mapwrite 0x025ceb thru 0x02900b (0x3321 bytes) 000721 1789266421.290384 punch from 0x011c60 to 0x01348b (0x182c bytes) 000723 1789266421.309293 fallocate 0x02b515 thru 0x02d843 (0x232f bytes) 000724 1789266421.319791 fallocate 0x015f81 thru 0x025dcd (0xfe4d bytes) 000725 1789266421.323778 trunc from 0x033281 to 0x0382e1 (0x5061 bytes) 000726 1789266421.328911 punch from 0x01cb71 to 0x02811d (0xb5ad bytes) 000727 1789266421.333436 mapread 0x028307 thru 0x029f42 (0x1c3c bytes) 000728 1789266421.339026 fallocate 0x03728c thru 0x03b635 (0x43aa bytes) 000729 1789266421.343928 trunc from 0x007443 to 0x010dc8 (0x9986 bytes) 000730 1789266421.347667 fallocate 0x01c4b6 thru 0x02a424 (0xdf6f bytes) 000731 1789266421.351750 mapread 0x0269b5 thru 0x0363bf (0xfa0b bytes) 000732 1789266421.377804 punch from 0x03709c to 0x03e397 (0x72fc bytes) 000733 1789266421.390271 MIRROR_RESYNC 16 mirrors 000734 1789266426.246545 write 0x01dc90 thru 0x02b9b9 (0xdd2a bytes) 000735 1789266426.301129 mapread 0x039f1e thru 0x03a893 (0x976 bytes) 000736 1789266426.311820 punch from 0x031b79 to 0x03f6ac (0xdb34 bytes) 000737 1789266426.319160 write 0x031b35 thru 0x03f966 (0xde32 bytes) 000738 1789266426.333001 MIRROR_RESYNC 16 mirrors 000739 1789266430.279928 trunc from 0x03f967 to 0x029a57 (0x15f0f bytes) 000740 1789266430.343707 fallocate 0x0140b5 thru 0x01c106 (0x8052 bytes) 000741 1789266430.351587 read 0x0157eb thru 0x01f3e8 (0x9bfe bytes) 000742 1789266430.377071 read 0x01abeb thru 0x020b67 (0x5f7d bytes) 000743 1789266430.413990 trunc from 0x016280 to 0x01a5b3 (0x4334 bytes) 000744 1789266430.421595 read 0x021eda thru 0x028dce (0x6ef5 bytes) 000745 1789266430.423656 read 0x001fc0 thru 0x0035ef (0x1630 bytes) 000746 1789266430.425629 mapread 0x01e7c3 thru 0x029a57 (0xb295 bytes) 000747 1789266430.427486 trunc from 0x029a58 to 0x01910e (0x10949 bytes) 000748 1789266430.473407 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1910f): Mirror 4: 0xf7e43b11 (stale) Mirror 5: 0xf7e43b11 (stale) Mirror 6: 0xf7e43b11 (stale) Mirror 11: 0xf7e43b11 (stale) Mirror 13: 0xf7e43b11 (stale) Mirror 14: 0xf7e43b11 (stale) Mirror 16: 0xf7e43b11 (stale) Mirror 17: 0xf7e43b11 (stale) Mirror 18: 0xf7e43b11 (stale) Mirror 20: 0xf7e43b11 (stale) Mirror 21: 0xf7e43b11 (stale) Mirror 22: 0xf7e43b11 (stale) Mirror 23: 0xf7e43b11 (stale) Mirror 24: 0xf7e43b11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1910f): Mirror 4: 0xf7e43b11 (stale) Mirror 5: 0xf7e43b11 (stale) Mirror 6: 0xf7e43b11 (stale) Mirror 11: 0xf7e43b11 (stale) Mirror 13: 0xf7e43b11 (stale) Mirror 14: 0xf7e43b11 (stale) Mirror 16: 0xf7e43b11 (stale) Mirror 17: 0xf7e43b11 (stale) Mirror 18: 0xf7e43b11 (stale) Mirror 20: 0xf7e43b11 (stale) Mirror 21: 0xf7e43b11 (stale) Mirror 22: 0xf7e43b11 (stale) Mirror 23: 0xf7e43b11 (stale) Mirror 24: 0xf7e43b11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1910f: skipped 000749 1789266431.226850 write 0x02d109 thru 0x039f2c (0xce24 bytes) 000751 1789266431.239517 trunc from 0x02066c to 0x021c92 (0x1627 bytes) 000752 1789266431.244183 fallocate 0x032ea0 thru 0x03c434 (0x9595 bytes) 000753 1789266431.253828 trunc from 0x039f2d to 0x01e958 (0x1b5d4 bytes) 000754 1789266431.290894 MIRROR_RESYNC 15 mirrors 000755 1789266433.134432 fallocate 0x01e9c7 thru 0x026b19 (0x8153 bytes) 000756 1789266433.180437 trunc from 0x01e959 to 0x0117ab (0xd1ad bytes) 000757 1789266433.200578 write 0x018da0 thru 0x021aea (0x8d4b bytes) 000758 1789266433.203629 read 0x01d9da thru 0x021aea (0x4111 bytes) 000759 1789266433.205277 mapread 0x016fb5 thru 0x01eda3 (0x7def bytes) 000760 1789266433.220948 mapwrite 0x0173d9 thru 0x01cb1b (0x5743 bytes) 000761 1789266433.258876 punch from 0x0180c6 to 0x0210c2 (0x8ffd bytes) 000762 1789266433.277108 MIRROR_RESYNC 15 mirrors 000763 1789266436.022236 fallocate 0x01c156 thru 0x01f4e5 (0x3390 bytes) 000764 1789266436.079284 mapread 0x01cbdc thru 0x021aea (0x4f0f bytes) 000765 1789266436.106612 write 0x01b288 thru 0x027abd (0xc836 bytes) 000766 1789266436.116325 MIRROR_RESYNC 15 mirrors 000768 1789266439.128399 mapwrite 0x01c42c thru 0x026930 (0xa505 bytes) 000769 1789266439.240325 write 0x03768e thru 0x03d647 (0x5fba bytes) 000770 1789266439.241690 read 0x038396 thru 0x03d647 (0x52b2 bytes) 000771 1789266439.243252 fallocate 0x00582d thru 0x013efe (0xe6d2 bytes) 000772 1789266439.251208 fallocate 0x00d006 thru 0x00d7f2 (0x7ed bytes) 000773 1789266439.258322 trunc from 0x02181d to 0x03002a (0xe80e bytes) 000774 1789266439.263609 write 0x01b274 thru 0x01b7aa (0x537 bytes) 000775 1789266439.266907 mapread 0x00418a thru 0x013b26 (0xf99d bytes) 000776 1789266439.286660 fallocate 0x036113 thru 0x037630 (0x151e bytes) 000777 1789266439.292310 punch from 0x00e723 to 0x00f77e (0x105c bytes) 000778 1789266439.297956 fallocate 0x00bd32 thru 0x014c12 (0x8ee1 bytes) 000779 1789266439.304710 read 0x027d47 thru 0x0357a4 (0xda5e bytes) 000781 1789266439.315816 read 0x01babf thru 0x020eb6 (0x53f8 bytes) 000783 1789266439.317584 trunc from 0x03d648 to 0x02edcb (0xe87c bytes) 000784 1789266439.342246 fallocate 0x00a332 thru 0x00be34 (0x1b03 bytes) 000785 1789266439.349314 read 0x023224 thru 0x02edcb (0xbba8 bytes) 000786 1789266439.350810 punch from 0x022ffb to 0x027e97 (0x4e9d bytes) 000787 1789266439.359763 fallocate 0x02a7b5 thru 0x035858 (0xb0a4 bytes) 000788 1789266439.366175 trunc from 0x02edcc to 0x0278e0 (0x74eb bytes) 000789 1789266439.390026 MIRROR_SPLIT mirror 10 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x278e1): Mirror 4: 0xf2e55a8a (stale) Mirror 5: 0xf2e55a8a (stale) Mirror 6: 0xf2e55a8a (stale) Mirror 11: 0xf2e55a8a (stale) Mirror 13: 0xf2e55a8a (stale) Mirror 14: 0xf2e55a8a (stale) Mirror 16: 0xf2e55a8a (stale) Mirror 17: 0xf2e55a8a (stale) Mirror 18: 0xf2e55a8a (stale) Mirror 21: 0xf2e55a8a (stale) Mirror 22: 0xf2e55a8a (stale) Mirror 23: 0xf2e55a8a (stale) Mirror 24: 0xf2e55a8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278e1): Mirror 4: 0xf2e55a8a (stale) Mirror 5: 0xf2e55a8a (stale) Mirror 6: 0xf2e55a8a (stale) Mirror 11: 0xf2e55a8a (stale) Mirror 13: 0xf2e55a8a (stale) Mirror 14: 0xf2e55a8a (stale) Mirror 16: 0xf2e55a8a (stale) Mirror 17: 0xf2e55a8a (stale) Mirror 18: 0xf2e55a8a (stale) Mirror 21: 0xf2e55a8a (stale) Mirror 22: 0xf2e55a8a (stale) Mirror 23: 0xf2e55a8a (stale) Mirror 24: 0xf2e55a8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278e1: skipped 000791 1789266440.094234 trunc from 0x0031b0 to 0x0063b9 (0x320a bytes) 000792 1789266440.110777 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x278e1): Mirror 4: 0xf2e55a8a (stale) Mirror 5: 0xf2e55a8a (stale) Mirror 6: 0xf2e55a8a (stale) Mirror 11: 0xf2e55a8a (stale) Mirror 13: 0xf2e55a8a (stale) Mirror 14: 0xf2e55a8a (stale) Mirror 16: 0xf2e55a8a (stale) Mirror 17: 0xf2e55a8a (stale) Mirror 21: 0xf2e55a8a (stale) Mirror 22: 0xf2e55a8a (stale) Mirror 23: 0xf2e55a8a (stale) Mirror 24: 0xf2e55a8a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x278e1): Mirror 4: 0xf2e55a8a (stale) Mirror 5: 0xf2e55a8a (stale) Mirror 6: 0xf2e55a8a (stale) Mirror 11: 0xf2e55a8a (stale) Mirror 13: 0xf2e55a8a (stale) Mirror 14: 0xf2e55a8a (stale) Mirror 16: 0xf2e55a8a (stale) Mirror 17: 0xf2e55a8a (stale) Mirror 21: 0xf2e55a8a (stale) Mirror 22: 0xf2e55a8a (stale) Mirror 23: 0xf2e55a8a (stale) Mirror 24: 0xf2e55a8a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x278e1: skipped 000793 1789266441.118896 trunc from 0x0278e1 to 0x0197fc (0xe0e4 bytes) 000794 1789266441.165303 read 0x01828e thru 0x0197fc (0x156f bytes) 000795 1789266441.176208 mapwrite 0x029d9e thru 0x03803b (0xe29e bytes) 000796 1789266441.293031 punch from 0x016f7c to 0x01d335 (0x63ba bytes) 000797 1789266441.317285 punch from 0x0181be to 0x020822 (0x8665 bytes) 000798 1789266441.321427 mapread 0x00faf3 thru 0x01a161 (0xa66f bytes) 000799 1789266441.352129 fallocate 0x006b0a thru 0x01238f (0xb886 bytes) 000800 1789266441.358171 read 0x004090 thru 0x00f2bf (0xb230 bytes) 000801 1789266441.379533 read 0x023884 thru 0x03009d (0xc81a bytes) 000802 1789266441.382837 punch from 0x00f702 to 0x0172ce (0x7bcd bytes) 000803 1789266441.388379 trunc from 0x0197e3 to 0x024e87 (0xb6a5 bytes) 000804 1789266441.395422 fallocate 0x02c990 thru 0x02dc40 (0x12b1 bytes) 000806 1789266441.404568 mapwrite 0x03be70 thru 0x03eaa6 (0x2c37 bytes) 000807 1789266441.462402 mapread 0x01cddc thru 0x020e88 (0x40ad bytes) 000808 1789266441.464160 fallocate 0x022443 thru 0x026d8a (0x4948 bytes) 000809 1789266441.470628 trunc from 0x03eaa7 to 0x003721 (0x3b385 bytes) 000811 1789266441.489113 write 0x024924 thru 0x033be2 (0xf2bf bytes) 000812 1789266441.493214 trunc from 0x033be3 to 0x0197e5 (0x1a3fd bytes) 000814 1789266441.510751 trunc from 0x00e593 to 0x011bcd (0x363b bytes) 000816 1789266441.524286 read 0x010c6c thru 0x0197e5 (0x8b7a bytes) 000817 1789266441.538699 read 0x007b4e thru 0x017222 (0xf6d5 bytes) 000818 1789266441.539435 fallocate 0x003298 thru 0x01068d (0xd3f6 bytes) 000819 1789266441.544274 punch from 0x000957 to 0x00ac2f (0xa2d9 bytes) 000820 1789266441.550832 write 0x02ded1 thru 0x03222a (0x435a bytes) 000821 1789266441.553212 mapwrite 0x036c1f thru 0x03c57b (0x595d bytes) 000822 1789266441.598054 trunc from 0x01b387 to 0x01dcf2 (0x296c bytes) 000823 1789266441.609480 MIRROR_RESYNC 13 mirrors 000824 1789266444.483162 mapread 0x0334ac thru 0x03c57b (0x90d0 bytes) 000825 1789266444.507032 mapwrite 0x00fb7f thru 0x01739e (0x7820 bytes) 000826 1789266444.611087 trunc from 0x03c57c to 0x01511b (0x27460 bytes) 000827 1789266444.653542 mapwrite 0x036f8d thru 0x03fee8 (0x8f5c bytes) 000828 1789266444.733148 trunc from 0x0002d3 to 0x00cfa3 (0xccd1 bytes) 000829 1789266444.739102 trunc from 0x03fee9 to 0x020386 (0x1fb62 bytes) 000830 1789266444.751925 write 0x00ef1d thru 0x012c94 (0x3d78 bytes) 000831 1789266444.755922 fallocate 0x031e3f thru 0x03ff0e (0xe0d0 bytes) 000832 1789266444.761485 write 0x02e717 thru 0x038b0c (0xa3f6 bytes) 000833 1789266444.763058 punch from 0x036f7c to 0x038b0c (0x1b91 bytes) 000834 1789266444.784242 punch from 0x026cac to 0x02d3f3 (0x6748 bytes) 000836 1789266444.789484 write 0x009d34 thru 0x0186aa (0xe977 bytes) 000837 1789266444.808587 read 0x02f540 thru 0x034f13 (0x59d4 bytes) 000838 1789266444.809620 read 0x00f339 thru 0x01278e (0x3456 bytes) 000839 1789266444.813294 punch from 0x01f02b to 0x02853d (0x9513 bytes) 000840 1789266444.821483 fallocate 0x031cb0 thru 0x0364da (0x482b bytes) 000841 1789266444.833858 MIRROR_RESYNC 13 mirrors 000842 1789266448.251786 fallocate 0x0030b5 thru 0x008f43 (0x5e8f bytes) 000843 1789266448.281665 write 0x03e6f5 thru 0x03ffff (0x190b bytes) 000844 1789266448.282767 trunc from 0x01447b to 0x0146e1 (0x267 bytes) 000845 1789266448.287589 mapwrite 0x0025b1 thru 0x00e4f4 (0xbf44 bytes) 000846 1789266448.323964 read 0x031068 thru 0x032996 (0x192f bytes) 000848 1789266448.329785 read 0x00da9a thru 0x00f18a (0x16f1 bytes) 000849 1789266448.330456 mapwrite 0x02f7d0 thru 0x037bd6 (0x8407 bytes) 000850 1789266448.356633 trunc from 0x02f3b8 to 0x03e41a (0xf063 bytes) 000851 1789266448.365595 MIRROR_SPLIT mirror 10 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x28cc3f5b (stale) Mirror 5: 0x28cc3f5b (stale) Mirror 6: 0x28cc3f5b (stale) Mirror 11: 0x28cc3f5b (stale) Mirror 13: 0x28cc3f5b (stale) Mirror 14: 0x28cc3f5b (stale) Mirror 16: 0x28cc3f5b (stale) Mirror 17: 0x28cc3f5b (stale) Mirror 21: 0x28cc3f5b (stale) Mirror 23: 0x28cc3f5b (stale) Mirror 24: 0x28cc3f5b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x28cc3f5b (stale) Mirror 5: 0x28cc3f5b (stale) Mirror 6: 0x28cc3f5b (stale) Mirror 11: 0x28cc3f5b (stale) Mirror 13: 0x28cc3f5b (stale) Mirror 14: 0x28cc3f5b (stale) Mirror 16: 0x28cc3f5b (stale) Mirror 17: 0x28cc3f5b (stale) Mirror 21: 0x28cc3f5b (stale) Mirror 23: 0x28cc3f5b (stale) Mirror 24: 0x28cc3f5b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000852 1789266449.005358 write 0x02c98e thru 0x03c63e (0xfcb1 bytes) 000853 1789266449.018745 mapread 0x03eefa thru 0x03ffff (0x1106 bytes) 000854 1789266449.026971 mapread 0x00fdc1 thru 0x01701c (0x725c bytes) 000856 1789266449.035879 write 0x007c0a thru 0x00e9e0 (0x6dd7 bytes) 000857 1789266449.040534 punch from 0x03ccf6 to 0x03ffff (0x330a bytes) 000858 1789266449.055189 fallocate 0x00c28d thru 0x010006 (0x3d7a bytes) 000859 1789266449.061091 punch from 0x02669e to 0x02a0bd (0x3a20 bytes) 000860 1789266449.064324 punch from 0x036af5 to 0x03b8e1 (0x4ded bytes) 000861 1789266449.069215 trunc from 0x040000 to 0x011cec (0x2e313 bytes) 000862 1789266449.093930 write 0x01e51a thru 0x0224ed (0x3fd4 bytes) 000864 1789266449.101539 mapread 0x0191e4 thru 0x01efa2 (0x5dbf bytes) 000865 1789266449.119386 trunc from 0x0224ee to 0x00aa4c (0x17aa1 bytes) 000866 1789266449.154173 mapread 0x0004ae thru 0x00aa4c (0xa59f bytes) 000867 1789266449.165142 trunc from 0x00aa4d to 0x01d07e (0x12632 bytes) 000868 1789266449.189300 mapread 0x00d456 thru 0x00ebfc (0x17a7 bytes) 000869 1789266449.205846 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d07f): Mirror 4: 0xb269d3a (stale) Mirror 5: 0xb269d3a (stale) Mirror 6: 0xb269d3a (stale) Mirror 11: 0xb269d3a (stale) Mirror 13: 0xb269d3a (stale) Mirror 14: 0xb269d3a (stale) Mirror 16: 0xb269d3a (stale) Mirror 17: 0xb269d3a (stale) Mirror 21: 0xb269d3a (stale) Mirror 23: 0xb269d3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d07f): Mirror 4: 0xb269d3a (stale) Mirror 5: 0xb269d3a (stale) Mirror 6: 0xb269d3a (stale) Mirror 11: 0xb269d3a (stale) Mirror 13: 0xb269d3a (stale) Mirror 14: 0xb269d3a (stale) Mirror 16: 0xb269d3a (stale) Mirror 17: 0xb269d3a (stale) Mirror 21: 0xb269d3a (stale) Mirror 23: 0xb269d3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d07f: skipped 000871 1789266449.805929 read 0x00dcfd thru 0x00ed2b (0x102f bytes) 000872 1789266449.813864 fallocate 0x00abe9 thru 0x00d3f0 (0x2808 bytes) 000873 1789266449.820800 fallocate 0x00891f thru 0x012805 (0x9ee7 bytes) 000874 1789266449.827446 write 0x02216a thru 0x025b53 (0x39ea bytes) 000875 1789266449.835278 write 0x019bb3 thru 0x01f672 (0x5ac0 bytes) 000876 1789266449.844821 punch from 0x02594f to 0x025b53 (0x205 bytes) 000877 1789266449.858118 mapwrite 0x001c7e thru 0x0085e5 (0x6968 bytes) 000878 1789266449.898494 fallocate 0x018dc1 thru 0x024ef2 (0xc132 bytes) 000879 1789266449.903758 mapread 0x00e2f3 thru 0x018c07 (0xa915 bytes) 000880 1789266449.923765 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x25b54): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b54): Mirror 4: 0x828acdd (stale) Mirror 5: 0x828acdd (stale) Mirror 6: 0x828acdd (stale) Mirror 11: 0x828acdd (stale) Mirror 13: 0x828acdd (stale) Mirror 14: 0x828acdd (stale) Mirror 16: 0x828acdd (stale) Mirror 17: 0x828acdd (stale) Mirror 21: 0x828acdd (stale) Mirror 23: 0x828acdd (stale) CRC-32 checksum value for chunk [0, 0x25b54): Mirror 1: 0xe731e46 Mirror 24: 0xe731e46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b54: skipped 000881 1789266450.979427 read 0x013730 thru 0x015d4f (0x2620 bytes) 000882 1789266450.988626 fallocate 0x00786e thru 0x00fd31 (0x84c4 bytes) 000883 1789266451.052676 mapwrite 0x039e59 thru 0x03ffff (0x61a7 bytes) 000884 1789266451.109044 write 0x0094b7 thru 0x012978 (0x94c2 bytes) 000885 1789266451.117058 trunc from 0x040000 to 0x03e88d (0x1772 bytes) 000886 1789266451.129682 trunc from 0x03e88e to 0x014d38 (0x29b55 bytes) 000887 1789266451.142154 read 0x013150 thru 0x014d38 (0x1be9 bytes) 000889 1789266451.150259 punch from 0x00cd64 to 0x00e3c9 (0x1666 bytes) 000890 1789266451.161913 write 0x03f5b7 thru 0x03ffff (0xa49 bytes) 000891 1789266451.163114 punch from 0x01be48 to 0x022c82 (0x6e3b bytes) 000892 1789266451.167011 read 0x013dba thru 0x021325 (0xd56c bytes) 000893 1789266451.174795 mapread 0x0265e7 thru 0x02df18 (0x7932 bytes) 000895 1789266451.176178 read 0x00dd6c thru 0x01b947 (0xdbdc bytes) 000896 1789266451.177375 read 0x00dab0 thru 0x017f65 (0xa4b6 bytes) 000897 1789266451.178240 mapwrite 0x004636 thru 0x00aef3 (0x68be bytes) 000898 1789266451.194120 trunc from 0x040000 to 0x006ff2 (0x3900d bytes) 000899 1789266451.215642 trunc from 0x006ff3 to 0x01b159 (0x14167 bytes) 000901 1789266451.229205 write 0x0277a0 thru 0x034163 (0xc9c4 bytes) 000902 1789266451.232115 mapread 0x00de08 thru 0x018e96 (0xb08f bytes) 000903 1789266451.240229 trunc from 0x023ce0 to 0x02493b (0xc5c bytes) 000904 1789266451.244223 fallocate 0x031a8b thru 0x03ffff (0xe575 bytes) 000905 1789266451.248317 mapwrite 0x037a53 thru 0x03ffff (0x85ad bytes) 000906 1789266451.272061 fallocate 0x027986 thru 0x02f2f9 (0x7974 bytes) 000908 1789266451.277977 mapread 0x002e6e thru 0x004425 (0x15b8 bytes) 000909 1789266451.279066 punch from 0x018fb7 to 0x019fcc (0x1016 bytes) 000910 1789266451.286472 MIRROR_RESYNC 12 mirrors 000911 1789266453.077167 write 0x0201d3 thru 0x02bdc7 (0xbbf5 bytes) 000913 1789266453.117670 read 0x006c29 thru 0x00e11f (0x74f7 bytes) 000914 1789266453.125055 trunc from 0x040000 to 0x01648b (0x29b74 bytes) 000915 1789266453.149139 read 0x00261e thru 0x00a75a (0x813d bytes) 000916 1789266453.157822 trunc from 0x00fcb2 to 0x01648b (0x67da bytes) 000917 1789266453.162182 mapread 0x0028d9 thru 0x0037d1 (0xef9 bytes) 000918 1789266453.163158 trunc from 0x00abd1 to 0x01648b (0xb8bb bytes) 000919 1789266453.167949 mapwrite 0x03455b thru 0x03ef4a (0xa9f0 bytes) 000920 1789266453.207160 trunc from 0x03ef4b to 0x02848e (0x16abc bytes) 000921 1789266453.229595 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b3:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2848f): Mirror 4: 0x887a9d82 (stale) Mirror 6: 0x887a9d82 (stale) Mirror 11: 0x887a9d82 (stale) Mirror 13: 0x887a9d82 (stale) Mirror 14: 0x887a9d82 (stale) Mirror 16: 0x887a9d82 (stale) Mirror 17: 0x887a9d82 (stale) Mirror 21: 0x887a9d82 (stale) Mirror 23: 0x887a9d82 (stale) Mirror 24: 0x887a9d82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2848f): Mirror 4: 0x887a9d82 (stale) Mirror 6: 0x887a9d82 (stale) Mirror 11: 0x887a9d82 (stale) Mirror 13: 0x887a9d82 (stale) Mirror 14: 0x887a9d82 (stale) Mirror 16: 0x887a9d82 (stale) Mirror 17: 0x887a9d82 (stale) Mirror 21: 0x887a9d82 (stale) Mirror 23: 0x887a9d82 (stale) Mirror 24: 0x887a9d82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2848f: skipped 000922 1789266453.885310 write 0x02332d thru 0x028488 (0x515c bytes) 000923 1789266453.901442 trunc from 0x02848f to 0x006b4e (0x21940 bytes) 000924 1789266453.922887 punch from 0x00666d to 0x006b4e (0x4e2 bytes) 000925 1789266453.936213 mapwrite 0x013351 thru 0x01c65b (0x930b bytes) 000926 1789266454.015617 trunc from 0x01c65c to 0x017c1b (0x4a40 bytes) 000927 1789266454.037662 trunc from 0x00961e to 0x010afd (0x74e0 bytes) 000928 1789266454.042233 mapread 0x016bd2 thru 0x017c1b (0x104a bytes) 000929 1789266454.043571 write 0x037f02 thru 0x03a759 (0x2858 bytes) 000930 1789266454.045028 mapread 0x02a9d9 thru 0x02be69 (0x1491 bytes) 000931 1789266454.052717 trunc from 0x02ecfc to 0x0301b0 (0x14b5 bytes) 000932 1789266454.057258 read 0x004000 thru 0x00d861 (0x9862 bytes) 000933 1789266454.065047 read 0x015134 thru 0x01ac5b (0x5b28 bytes) 000934 1789266454.083940 MIRROR_RESYNC 11 mirrors 000935 1789266456.526531 mapread 0x003561 thru 0x005e9d (0x293d bytes) 000936 1789266456.562607 trunc from 0x03a75a to 0x03b89b (0x1142 bytes) 000937 1789266456.614726 MIRROR_RESYNC 11 mirrors 000938 1789266459.467092 mapread 0x02e931 thru 0x030a5b (0x212b bytes) 000939 1789266459.491932 read 0x01b472 thru 0x023e3f (0x89ce bytes) 000940 1789266459.505522 punch from 0x0361b1 to 0x039e52 (0x3ca2 bytes) 000941 1789266459.551719 write 0x02389f thru 0x029858 (0x5fba bytes) 000942 1789266459.564262 trunc from 0x03b89c to 0x02d478 (0xe423 bytes) 000943 1789266459.580855 fallocate 0x03cdd1 thru 0x03ffff (0x322f bytes) 000944 1789266459.584679 read 0x02bdc3 thru 0x02d478 (0x16b6 bytes) 000947 1789266459.595472 mapwrite 0x0310bd thru 0x038a70 (0x79b4 bytes) 000948 1789266459.702684 punch from 0x032b94 to 0x033fda (0x1447 bytes) 000951 1789266459.715427 mapwrite 0x013f12 thru 0x015e71 (0x1f60 bytes) 000952 1789266459.740957 read 0x01cbc4 thru 0x02c43a (0xf877 bytes) 000953 1789266459.742477 mapread 0x0012b5 thru 0x00a42e (0x917a bytes) 000954 1789266459.749672 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x38a71): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a71): Mirror 4: 0x5470207 (stale) Mirror 6: 0x5470207 (stale) Mirror 11: 0x5470207 (stale) Mirror 13: 0x5470207 (stale) Mirror 14: 0x5470207 (stale) Mirror 16: 0x5470207 (stale) Mirror 17: 0x5470207 (stale) Mirror 21: 0x5470207 (stale) Mirror 23: 0x5470207 (stale) Mirror 24: 0x5470207 (stale) CRC-32 checksum value for chunk [0, 0x38a71): Mirror 1: 0xab52ec3e Mirror 25: 0xab52ec3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a71: skipped 000955 1789266460.893736 fallocate 0x021195 thru 0x028ba8 (0x7a14 bytes) 000958 1789266460.950640 fallocate 0x017cd0 thru 0x01a558 (0x2889 bytes) 000959 1789266460.955468 mapread 0x0136fc thru 0x021759 (0xe05e bytes) 000960 1789266460.982392 mapread 0x027370 thru 0x027e3b (0xacc bytes) 000961 1789266460.992702 mapread 0x031490 thru 0x038a70 (0x75e1 bytes) 000962 1789266461.004847 trunc from 0x02735d to 0x02ff7f (0x8c23 bytes) 000963 1789266461.020576 trunc from 0x02a7e4 to 0x030cd6 (0x64f3 bytes) 000964 1789266461.035198 mapread 0x00a1b8 thru 0x012ccf (0x8b18 bytes) 000965 1789266461.053951 mapread 0x025446 thru 0x02db0b (0x86c6 bytes) 000966 1789266461.075190 mapread 0x015cb7 thru 0x023aab (0xddf5 bytes) 000967 1789266461.076984 trunc from 0x038a71 to 0x004b82 (0x33eee bytes) 000968 1789266461.123692 write 0x03f90d thru 0x03ffff (0x6f3 bytes) 000969 1789266461.125017 trunc from 0x040000 to 0x01c7e8 (0x23817 bytes) 000970 1789266461.144864 trunc from 0x005ff2 to 0x00c176 (0x6185 bytes) 000971 1789266461.150999 trunc from 0x01c7e9 to 0x02e747 (0x11f5f bytes) 000972 1789266461.169021 write 0x00ac56 thru 0x0126be (0x7a69 bytes) 000974 1789266461.184037 write 0x0352bf thru 0x03ffff (0xad41 bytes) 000975 1789266461.186781 trunc from 0x040000 to 0x002af7 (0x3d508 bytes) 000976 1789266461.209530 punch from 0x001951 to 0x002af7 (0x11a7 bytes) 000977 1789266461.215027 trunc from 0x000c53 to 0x002af7 (0x1ea5 bytes) 000978 1789266461.220005 trunc from 0x002af8 to 0x00b8e3 (0x8dec bytes) 000979 1789266461.241981 punch from 0x009e4c to 0x00b8e3 (0x1a98 bytes) 000980 1789266461.249107 mapwrite 0x02ba59 thru 0x030c79 (0x5221 bytes) 000982 1789266461.316694 write 0x02c092 thru 0x02c9e5 (0x954 bytes) 000983 1789266461.318606 mapwrite 0x016262 thru 0x021bc4 (0xb963 bytes) 000984 1789266461.361034 write 0x00fbb5 thru 0x015472 (0x58be bytes) 000985 1789266461.362772 trunc from 0x004614 to 0x010e81 (0xc86e bytes) 000986 1789266461.375756 write 0x02d01a thru 0x0339a1 (0x6988 bytes) 000987 1789266461.376977 mapwrite 0x031c27 thru 0x032ac9 (0xea3 bytes) 000988 1789266461.398821 MIRROR_RESYNC 12 mirrors 000989 1789266464.524553 write 0x03f36b thru 0x03ffff (0xc95 bytes) 000990 1789266464.584002 trunc from 0x040000 to 0x01ee7f (0x21180 bytes) 000991 1789266464.618678 trunc from 0x01dbdd to 0x01ee7f (0x12a3 bytes) 000993 1789266464.628034 trunc from 0x01ee80 to 0x01b7ed (0x3692 bytes) 000994 1789266464.645822 fallocate 0x02c7ec thru 0x03b370 (0xeb85 bytes) 000995 1789266464.652816 trunc from 0x01b7ee to 0x03266e (0x16e81 bytes) 000997 1789266464.675227 trunc from 0x03266f to 0x019e0f (0x1885f bytes) 000999 1789266464.697764 mapread 0x011eb9 thru 0x019e0f (0x7f57 bytes) 001000 1789266464.719961 mapread 0x00e06f thru 0x0105be (0x2550 bytes) All operations completed A-OK! PASS 70a (135s) == sanity-flr test 71: check mirror extend parent fid ==== 22:27:49 (1789266469) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1e4:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 22:27:59 (1789266479) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174529 s, 12.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131166 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 22:28:08 (1789266488) Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4752640 .. Extending file size to 4069440 .. failed Extending file size to 7363552 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5100096 .. Extending file size to 5360192 .. Extending file size to 6861856 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7227456 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8375456 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8504800 .. Extending file size to 7592128 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 7886624 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8133248 .. Extending file size to 8819648 .. Extending file size to 8209248 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8330848 .. Waiting 61367 61369 61370 61371 61372 done 192.168.204.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg407-client.virtnet /mnt/lustre2 (opts:) 192.168.204.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg407-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:29:29 (1789266569) Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2747680 .. Extending file size to 1859104 .. failed Extending file size to 4707744 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7689920 .. Extending file size to 6801536 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8496640 .. Extending file size to 7302176 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8081760 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8511904 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8639136 .. Extending file size to 8832192 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8639648 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63112 63113 63116 63117 63119 done 192.168.204.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg407-client.virtnet /mnt/lustre2 (opts:) 192.168.204.107@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg407-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:30:48 (1789266648) Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2304 1269000 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2368 1268936 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25104 3581916 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17556 3589464 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42660 7171380 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2304 1269000 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2368 1268936 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25104 3581916 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17556 3589464 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42660 7171380 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142045 s, 14.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173824 s, 12.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x35dacac0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg407-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:31:06 (1789266666) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0735738 s, 28.5 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 22:31:12 (1789266672) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x60:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:31:18 (1789266678) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118697 s, 17.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x254:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x254:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x255:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:31:25 (1789266685) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0863809 s, 24.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:31:31 (1789266691) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0828792 s, 25.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x255:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x256:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x255:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x256:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x257:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:32:02 (1789266722) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg407-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798075 s, 13.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x256:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x259:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x258:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x256:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x259:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x258:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x257:0x0] } PASS 204d (46s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:32:48 (1789266768) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg407-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111725 s, 18.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (46s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:33:34 (1789266814) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg407-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg407-server: Pool lustre.test_204f created Adding targets to pool oleg407-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg407-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120712 s, 17.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg407-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg407-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg407-server: Pool lustre.test_204f destroyed PASS 204f (68s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:34:43 (1789266883) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:34:49 (1789266889) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:34:56 (1789266896) oleg407-server: Pool lustre.test_206 created Adding targets to pool oleg407-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg407-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg407-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg407-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg407-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:35:22 (1789266922) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599831 s, 17.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303747 s, 34.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1f4bf743 Mirror 3: 0x1f4bf743 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:35:31 (1789266931) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:35:35 (1789266935) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:35:38 (1789266938) Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0662969 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1cc0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1750000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1750000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1750000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x25c0000 Read this (4 bytes): BBBB 192.168.204.107@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg407-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:36:16 (1789266976) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0321795 s, 255 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:36:26 (1789266986) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111922 s, 9.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-12 22:36:28.000000000 -0400 Modify: 2026-09-12 22:36:28.000000000 -0400 Change: 2026-09-12 22:36:29.000000000 -0400 Birth: 2026-09-12 22:36:27.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:36:35 (1789266995) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196074 s, 53.5 MB/s oleg407-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg407-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg407-server: O/280000401/d25/139: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:36:56 (1789267016) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.261431 s, 157 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0417166 s, 98.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 12 22:36 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 12 22:36 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:37:05 (1789267025) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:37:13 (1789267033) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1770 sec =========== 22:37:22 (1789267042) === sanity-flr: start cleanup 22:37:23 (1789267043) === === sanity-flr: finish cleanup 22:37:31 (1789267051) ===