-----============= acceptance-small: sanity-flr ============----- Wed Sep 16 14:24:29 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg234-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 14:24:50 (1789583090) === oleg234-client.virtnet: executing check_config_client /mnt/lustre oleg234-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg234-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93bf4843f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93bf4843f800.idle_timeout=debug disable quota as required oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 14:25:40 (1789583140) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8234] == sanity-flr test 0a: lfs mirror create with -N option == 14:25:42 (1789583142) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:26:02 (1789583162) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:26:06 (1789583166) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:26:10 (1789583170) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:26:38 (1789583198) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:26:42 (1789583202) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 14:26:47 (1789583207) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93bf4843f800.stats=clear osc.lustre-OST0001-osc-ffff93bf4843f800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:26:56 (1789583216) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:27:09 (1789583229) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:27:17 (1789583237) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:27:21 (1789583241) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:27:29 (1789583249) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 14:27:39 (1789583259) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:27:47 (1789583267) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:27:57 (1789583277) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (12s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:28:09 (1789583289) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:28:23 (1789583303) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:28:30 (1789583310) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.54105 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128124 s, 8.2 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:28:43 (1789583323) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.594183 s, 26.5 MB/s PASS 22 (13s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:28:56 (1789583336) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:29:09 (1789583349) Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93bf4843f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93bf4843f800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93bf4843f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93bf4843f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:30:25 (1789583425) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0881295 s, 23.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93bf4843f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93bf4843f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:31:12 (1789583472) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93bf4843f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93bf4843f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93bf4843f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93bf4843f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:32:39 (1789583559) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.11416 s, 16.8 MB/s -rw-r--r-- 1 root root 52428800 Sep 16 14:32 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.903s user 0m0.001s sys 0m0.197s Stopping /mnt/lustre-ost1 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93bf4843f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93bf4843f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.136s user 0m0.000s sys 0m0.176s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg234-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg234-server oleg234-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93bf4843f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93bf4843f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.396s user 0m0.001s sys 0m0.190s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg234-server: oleg234-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg234-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg234-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93bf4843f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93bf4843f800.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:34:21 (1789583661) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:34:24 (1789583664) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:34:29 (1789583669) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 14:34:33 (1789583673) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 14:34:42 (1789583682) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.56341 s, 22.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:34:56 (1789583696) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267695 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304283 s, 34.5 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:35:05 (1789583705) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:35:15 (1789583715) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0019415 s, 527 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00142648 s, 718 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00175622 s, 583 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000936326 s, 1.1 MB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 14:35:27 (1789583727) create test files with size 15757603 .. 09d3d6d9b587b6a675cf6d154e02eb0f - ee280d3f4f3e41cec9f34e808cd227c6 - 0808b056e7ef062d5c9dbfdc412b922d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 14:35:53 (1789583753) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:36:12 (1789583772) PASS 39 (10s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:36:21 (1789583781) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952797 s, 11.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721722 s, 14.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 14:36:33 (1789583793) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104342 s, 20.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.455266 s, 9.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.212732 s, 9.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 14:36:53 (1789583813) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 14:36:58 (1789583818) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:37:03 (1789583823) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358615 s, 29.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 14:37:13 (1789583833) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.38188 s, 8.2 MB/s PASS 44a (13s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:37:27 (1789583847) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:37:39 (1789583859) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.455002 s, 23.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:37:54 (1789583874) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff93bf58a1c000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.486151 s, 21.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg234-client.virtnet /mnt/lustre (opts:) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff93bf42d99800.xattr_cache=1 PASS 44e (21s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:38:16 (1789583896) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (51s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:39:06 (1789583946) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (46s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:39:52 (1789583992) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 14:39:57 (1789583997) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.225402 s, 14.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.180256 s, 17.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x2613d148 (nosync,stale) Mirror 2: 0x2613d148 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 883211a1a7f1b9dc7baac69a8d17aaa1 - mirror 1 checksum: c03ce25c3f7e56f20798511d8024c954 - mirror 2 checksum: 883211a1a7f1b9dc7baac69a8d17aaa1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-16T14:40:00-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-16T14:40:00-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 883211a1a7f1b9dc7baac69a8d17aaa1 - mirror 1 checksum: c03ce25c3f7e56f20798511d8024c954 - mirror 2 checksum: c03ce25c3f7e56f20798511d8024c954 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-16T14:40:00-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-16T14:40:00-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 48 (14s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 14:40:11 (1789584011) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656571 s, 16.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (11s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:40:22 (1789584022) Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 14:40:35 (1789584035) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 14:41:07 (1789584067) targets are mounted 14:41:07 (1789584067) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472852 s, 22.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:41:29 (1789584089) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.086604 s, 236 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0245519 s, 834 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (14s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:41:43 (1789584103) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0525671 s, 390 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00977709 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:41:55 (1789584115) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00797261 s, 2.1 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:42:04 (1789584124) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0623662 s, 328 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:42:13 (1789584133) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0263775 s, 155 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0625123 s, 65.5 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:42:21 (1789584141) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:42:46 (1789584166) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:43:15 (1789584195) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:43:41 (1789584221) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:44:05 (1789584245) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 14:44:13 (1789584253) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57603 mirror split pid 57604 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57603 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57604 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 14:45:21 (1789584321) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4323 000001 1789584323.702984 MIRROR_EXTEND to 2 mirrors skipping zero size read skipping zero size read 000005 1789584324.017291 mapwrite 0x00d504 thru 0x016889 (0x9386 bytes) 000007 1789584324.178402 trunc from 0x0013d9 to 0x007fec (0x6c14 bytes) 000008 1789584324.189263 read 0x003cb6 thru 0x0111aa (0xd4f5 bytes) 000009 1789584324.213748 mapwrite 0x0054e0 thru 0x009945 (0x4466 bytes) 000010 1789584324.242018 read 0x0091f5 thru 0x01270e (0x951a bytes) truncating to largest ever: 0x303cc 000011 1789584324.242757 trunc from 0x01688a to 0x0303cb (0x19b42 bytes) 000012 1789584324.273072 mapread 0x02f0a8 thru 0x0303cb (0x1324 bytes) 000013 1789584324.289745 mapread 0x0142cb thru 0x01ad39 (0x6a6f bytes) 000014 1789584324.292933 mapwrite 0x0065f0 thru 0x010e41 (0xa852 bytes) 000015 1789584324.355136 write 0x031a45 thru 0x039839 (0x7df5 bytes) 000016 1789584324.356702 punch from 0x023309 to 0x02cdc5 (0x9abd bytes) 000017 1789584324.374623 punch from 0x01f030 to 0x029cfb (0xaccc bytes) 000018 1789584324.386013 read 0x027b62 thru 0x02ba86 (0x3f25 bytes) 000019 1789584324.401521 fallocate 0x014cab thru 0x01dbad (0x8f03 bytes) truncating to largest ever: 0x38065 000020 1789584324.409020 trunc from 0x03983a to 0x038064 (0x17d5 bytes) 000021 1789584324.436847 fallocate 0x02e679 thru 0x035798 (0x7120 bytes) 000022 1789584324.440736 punch from 0x017efe to 0x0228c1 (0xa9c4 bytes) 000023 1789584324.444817 mapwrite 0x01d13e thru 0x01dda7 (0xc6a bytes) 000024 1789584324.484913 read 0x0007ee thru 0x005bd5 (0x53e8 bytes) 000025 1789584324.495568 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000028 1789584324.632997 write 0x02802f thru 0x03099f (0x8971 bytes) 000029 1789584324.653591 trunc from 0x0012fb to 0x003069 (0x1d6f bytes) 000030 1789584324.659638 mapwrite 0x01918d thru 0x021dc5 (0x8c39 bytes) 000031 1789584324.717913 read 0x036386 thru 0x038064 (0x1cdf bytes) fallocating to largest ever: 0x39edf 000032 1789584324.726182 fallocate 0x02eda7 thru 0x039ede (0xb138 bytes) 000033 1789584324.738052 trunc from 0x000206 to 0x00dde1 (0xdbdc bytes) 000034 1789584324.751898 trunc from 0x01a031 to 0x01bdab (0x1d7b bytes) 000035 1789584324.767878 read 0x036973 thru 0x039ede (0x356c bytes) 000036 1789584324.796079 fallocate 0x0027dc thru 0x0107d2 (0xdff7 bytes) 000037 1789584324.803582 fallocate 0x002af6 thru 0x003544 (0xa4f bytes) 000038 1789584324.815614 read 0x028840 thru 0x0309b2 (0x8173 bytes) 000039 1789584324.816626 trunc from 0x039edf to 0x00aadf (0x2f3ff bytes) 000040 1789584324.840588 read 0x0007f8 thru 0x00aadf (0xa2e8 bytes) 000041 1789584324.845976 MIRROR_EXTEND to 2 mirrors 000042 1789584325.570906 mapwrite 0x038e76 thru 0x03c9d5 (0x3b60 bytes) 000043 1789584325.643438 mapwrite 0x0121d4 thru 0x018e5b (0x6c88 bytes) 000044 1789584325.680913 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c9d6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9d6): Mirror 2: 0x49f5cbee (stale) CRC-32 checksum value for chunk [0, 0x3c9d6): Mirror 1: 0xfcaf3d10 Mirror 3: 0xfcaf3d10 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9d6: skipped 000045 1789584326.566854 mapwrite 0x029c9d thru 0x02d667 (0x39cb bytes) 000047 1789584326.684650 mapwrite 0x00bf67 thru 0x017db8 (0xbe52 bytes) 000048 1789584326.779411 punch from 0x00815c to 0x00b693 (0x3538 bytes) 000049 1789584326.787033 mapwrite 0x01e763 thru 0x01f725 (0xfc3 bytes) 000050 1789584326.830680 read 0x009279 thru 0x01377f (0xa507 bytes) 000051 1789584326.842237 mapwrite 0x006899 thru 0x00f98e (0x90f6 bytes) 000052 1789584326.888042 read 0x001a44 thru 0x00dad4 (0xc091 bytes) 000055 1789584326.895520 mapwrite 0x032839 thru 0x0336ca (0xe92 bytes) 000056 1789584326.923627 write 0x009f4d thru 0x010497 (0x654b bytes) 000058 1789584326.930703 write 0x025aab thru 0x03582e (0xfd84 bytes) 000059 1789584326.932680 trunc from 0x03836b to 0x03c9d5 (0x466b bytes) 000060 1789584326.937312 trunc from 0x00be1b to 0x011c56 (0x5e3c bytes) 000061 1789584326.957987 read 0x01c50c thru 0x02a83b (0xe330 bytes) 000062 1789584326.958859 mapread 0x011d27 thru 0x0149d6 (0x2cb0 bytes) 000063 1789584326.973234 fallocate 0x009425 thru 0x00aa38 (0x1614 bytes) 000064 1789584326.978582 trunc from 0x03c9d6 to 0x01d967 (0x1f06e bytes) 000065 1789584327.045558 mapread 0x00972b thru 0x0177a1 (0xe077 bytes) 000066 1789584327.049888 write 0x01f3a7 thru 0x02182c (0x2486 bytes) 000067 1789584327.055042 mapwrite 0x013c2b thru 0x016715 (0x2aeb bytes) 000068 1789584327.071437 mapwrite 0x017c19 thru 0x01ca48 (0x4e30 bytes) 000069 1789584327.098529 mapread 0x00ca91 thru 0x014182 (0x76f2 bytes) 000070 1789584327.099750 mapwrite 0x0256e5 thru 0x0291b1 (0x3acd bytes) 000071 1789584327.168974 mapread 0x0174e5 thru 0x01aafe (0x361a bytes) 000072 1789584327.169999 mapwrite 0x013104 thru 0x020795 (0xd692 bytes) 000073 1789584327.226699 trunc from 0x00801c to 0x0152cf (0xd2b4 bytes) 000074 1789584327.235905 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x291b2): Mirror 3: 0x84ad92f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x291b2): Mirror 3: 0x84ad92f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x291b2: skipped 000075 1789584327.568242 punch from 0x0260b8 to 0x0291b1 (0x30fa bytes) 000076 1789584327.581557 MIRROR_RESYNC 2 mirrors 000079 1789584328.351923 read 0x01d2f3 thru 0x01dee0 (0xbee bytes) 000080 1789584328.355985 mapread 0x02794c thru 0x0291b1 (0x1866 bytes) 000081 1789584328.372333 trunc from 0x0291b2 to 0x00055e (0x28c53 bytes) 000082 1789584328.405276 trunc from 0x0001e3 to 0x00055e (0x37c bytes) 000084 1789584328.411571 trunc from 0x0001d9 to 0x00055e (0x386 bytes) 000085 1789584328.417141 trunc from 0x0002ed to 0x00055e (0x272 bytes) 000086 1789584328.421997 fallocate 0x02ef51 thru 0x03372a (0x47da bytes) 000087 1789584328.426564 read 0x000c56 thru 0x00ca47 (0xbdf2 bytes) 000088 1789584328.438211 trunc from 0x03372b to 0x01d721 (0x16009 bytes) 000089 1789584328.459559 trunc from 0x01122e to 0x014769 (0x353c bytes) 000090 1789584328.464853 punch from 0x006237 to 0x0089bc (0x2786 bytes) 000091 1789584328.469693 punch from 0x009a2f to 0x0192d0 (0xf8a2 bytes) 000092 1789584328.475372 punch from 0x00a41c to 0x016a82 (0xc667 bytes) 000093 1789584328.480266 trunc from 0x01d722 to 0x0231c5 (0x5aa4 bytes) 000094 1789584328.504782 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000095 1789584328.711753 mapwrite 0x01e4cd thru 0x02c419 (0xdf4d bytes) 000096 1789584328.803552 write 0x0243b0 thru 0x032d34 (0xe985 bytes) 000098 1789584328.811126 read 0x00d5ef thru 0x013342 (0x5d54 bytes) 000099 1789584328.819849 read 0x0056f3 thru 0x006e9d (0x17ab bytes) 000100 1789584328.827992 fallocate 0x0304e3 thru 0x0343a0 (0x3ebe bytes) 000101 1789584328.833044 mapwrite 0x02a47c thru 0x03500d (0xab92 bytes) 000102 1789584328.871229 mapwrite 0x02a267 thru 0x02d2bf (0x3059 bytes) 000103 1789584328.883383 mapwrite 0x0094ac thru 0x00d3d4 (0x3f29 bytes) 000104 1789584328.897424 punch from 0x02440a to 0x02545b (0x1052 bytes) 000106 1789584328.904947 punch from 0x027e59 to 0x03500d (0xd1b5 bytes) 000107 1789584328.910949 trunc from 0x03500e to 0x002da3 (0x3226a bytes) 000108 1789584328.930972 MIRROR_EXTEND to 2 mirrors 000109 1789584329.656977 trunc from 0x002159 to 0x002da3 (0xc4b bytes) 000110 1789584329.694927 write 0x03fb39 thru 0x03ffff (0x4c7 bytes) 000111 1789584329.705557 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000112 1789584329.859375 trunc from 0x040000 to 0x004c46 (0x3b3b9 bytes) 000115 1789584329.892587 read 0x00034c thru 0x004c46 (0x48fb bytes) 000116 1789584329.903569 punch from 0x001a1b to 0x004c46 (0x322c bytes) 000117 1789584329.911387 read 0x004529 thru 0x004c46 (0x71e bytes) fallocating to largest ever: 0x40000 000118 1789584329.917303 fallocate 0x03b4f6 thru 0x03ffff (0x4b0a bytes) 000119 1789584329.923259 trunc from 0x004c47 to 0x00b47d (0x6837 bytes) 000120 1789584329.948853 punch from 0x00a8a1 to 0x00b47d (0xbdd bytes) 000122 1789584329.956500 write 0x03a457 thru 0x03ffff (0x5ba9 bytes) 000123 1789584329.959585 write 0x000e0a thru 0x00bca5 (0xae9c bytes) 000124 1789584329.965770 read 0x037d1e thru 0x03ffff (0x82e2 bytes) 000125 1789584329.976221 punch from 0x011e14 to 0x0198cf (0x7abc bytes) 000126 1789584329.980997 trunc from 0x040000 to 0x03f500 (0xaff bytes) 000127 1789584330.000666 write 0x0074e1 thru 0x011b69 (0xa689 bytes) 000128 1789584330.019061 write 0x037df3 thru 0x03b50e (0x371c bytes) 000129 1789584330.020622 read 0x024cda thru 0x02ceef (0x8216 bytes) 000131 1789584330.027346 punch from 0x00ad76 to 0x0181d4 (0xd45f bytes) 000133 1789584330.050475 trunc from 0x02a36a to 0x030eec (0x6b83 bytes) 000134 1789584330.059847 trunc from 0x01161e to 0x012e47 (0x182a bytes) 000135 1789584330.065565 write 0x034be2 thru 0x03ffff (0xb41e bytes) 000136 1789584330.071276 write 0x01c8c1 thru 0x02a720 (0xde60 bytes) 000137 1789584330.094491 mapread 0x0275ff thru 0x029a95 (0x2497 bytes) 000138 1789584330.095696 write 0x030bc6 thru 0x03a075 (0x94b0 bytes) 000139 1789584330.110916 read 0x004d7b thru 0x00e714 (0x999a bytes) 000140 1789584330.115009 read 0x03e18f thru 0x03ffff (0x1e71 bytes) 000141 1789584330.120306 punch from 0x0038de to 0x008277 (0x499a bytes) 000142 1789584330.137002 trunc from 0x040000 to 0x006b69 (0x39496 bytes) 000144 1789584330.162213 fallocate 0x005181 thru 0x008acd (0x394d bytes) 000145 1789584330.168215 write 0x0396df thru 0x03ffff (0x6921 bytes) 000146 1789584330.170712 trunc from 0x040000 to 0x027de1 (0x1821e bytes) 000147 1789584330.184020 trunc from 0x027de2 to 0x034852 (0xca71 bytes) 000148 1789584330.200449 read 0x0264a0 thru 0x02f212 (0x8d73 bytes) 000149 1789584330.229364 trunc from 0x02ef0c to 0x0321c0 (0x32b5 bytes) 000151 1789584330.237250 punch from 0x007c2a to 0x012870 (0xac47 bytes) 000152 1789584330.242786 trunc from 0x00ee6e to 0x014158 (0x52eb bytes) 000153 1789584330.247313 fallocate 0x02e570 thru 0x033d16 (0x57a7 bytes) 000154 1789584330.252797 fallocate 0x0044ab thru 0x00dd3d (0x9893 bytes) 000155 1789584330.257868 trunc from 0x00ea58 to 0x0138eb (0x4e94 bytes) 000156 1789584330.262866 punch from 0x02a093 to 0x02d36f (0x32dd bytes) 000157 1789584330.269911 write 0x02a0ff thru 0x032393 (0x8295 bytes) 000158 1789584330.294291 MIRROR_EXTEND to 2 mirrors 000159 1789584330.892874 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000160 1789584331.022826 trunc from 0x02d205 to 0x034083 (0x6e7f bytes) 000161 1789584331.046679 fallocate 0x017ad8 thru 0x0191ca (0x16f3 bytes) 000162 1789584331.051564 trunc from 0x023d29 to 0x030f08 (0xd1e0 bytes) 000163 1789584331.056611 read 0x010670 thru 0x017ba4 (0x7535 bytes) 000164 1789584331.066500 mapwrite 0x03d186 thru 0x03ffff (0x2e7a bytes) 000165 1789584331.202013 trunc from 0x02ec23 to 0x03987d (0xac5b bytes) 000166 1789584331.213236 trunc from 0x040000 to 0x00aa2f (0x355d0 bytes) 000167 1789584331.241583 MIRROR_EXTEND to 2 mirrors 000168 1789584331.728074 trunc from 0x000761 to 0x004803 (0x40a3 bytes) 000169 1789584331.775475 punch from 0x001517 to 0x0054c3 (0x3fad bytes) 000171 1789584331.783851 mapread 0x00706f thru 0x00aa2f (0x39c1 bytes) 000172 1789584331.810588 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xaa30): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaa30): Mirror 2: 0x593ea609 (stale) CRC-32 checksum value for chunk [0, 0xaa30): Mirror 1: 0x210b2812 Mirror 3: 0x210b2812 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaa30: skipped 000173 1789584333.039867 mapwrite 0x0047cd thru 0x01350a (0xed3e bytes) 000174 1789584333.212352 fallocate 0x02eeb4 thru 0x0303b6 (0x1503 bytes) 000175 1789584333.219933 read 0x009019 thru 0x00aa5c (0x1a44 bytes) 000176 1789584333.222469 mapwrite 0x012b86 thru 0x01b4ab (0x8926 bytes) 000177 1789584333.308346 fallocate 0x0065a6 thru 0x014a86 (0xe4e1 bytes) 000178 1789584333.317400 MIRROR_RESYNC 3 mirrors 000179 1789584334.362552 punch from 0x009868 to 0x00e6f7 (0x4e90 bytes) 000180 1789584334.422475 trunc from 0x01b4ac to 0x037270 (0x1bdc5 bytes) 000181 1789584334.481183 read 0x027ac3 thru 0x037270 (0xf7ae bytes) 000182 1789584334.508606 trunc from 0x037271 to 0x004096 (0x331da bytes) 000183 1789584334.537645 fallocate 0x00ec63 thru 0x00f254 (0x5f2 bytes) 000184 1789584334.544932 mapwrite 0x00de3c thru 0x015757 (0x791c bytes) 000185 1789584334.596448 read 0x008019 thru 0x013468 (0xb450 bytes) 000186 1789584334.598487 trunc from 0x00c60a to 0x015757 (0x914e bytes) 000187 1789584334.607627 fallocate 0x03e93f thru 0x03ffff (0x16c1 bytes) 000188 1789584334.613563 mapread 0x0326d8 thru 0x036d77 (0x46a0 bytes) 000189 1789584334.669621 fallocate 0x00f60b thru 0x01d610 (0xe006 bytes) 000190 1789584334.677045 read 0x036b60 thru 0x03ffff (0x94a0 bytes) 000192 1789584334.690640 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8236f91b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8236f91b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000193 1789584334.956031 mapwrite 0x011bbe thru 0x016d26 (0x5169 bytes) 000194 1789584335.001632 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8236f91b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x61a33470 Mirror 4: 0x61a33470 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000195 1789584336.386786 punch from 0x00c4b4 to 0x01b844 (0xf391 bytes) 000196 1789584336.456159 trunc from 0x040000 to 0x02394b (0x1c6b4 bytes) 000197 1789584336.513129 write 0x012c0f thru 0x01408b (0x147d bytes) 000198 1789584336.537972 write 0x02a367 thru 0x02ac89 (0x923 bytes) 000199 1789584336.539045 trunc from 0x0173cb to 0x027040 (0xfc76 bytes) 000200 1789584336.545819 trunc from 0x015493 to 0x01e1c3 (0x8d31 bytes) 000201 1789584336.567261 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ac8a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ac8a): Mirror 3: 0x62acafbf (stale) Mirror 4: 0xa2f0c8f7 (stale) CRC-32 checksum value for chunk [0, 0x2ac8a): Mirror 1: 0x31966434 Mirror 5: 0x31966434 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ac8a: skipped 000202 1789584338.207948 mapread 0x017827 thru 0x0247d0 (0xcfaa bytes) 000203 1789584338.339193 trunc from 0x02ac8a to 0x00a854 (0x20435 bytes) 000204 1789584338.459401 fallocate 0x02b00e thru 0x039370 (0xe363 bytes) 000205 1789584338.470998 write 0x026cb3 thru 0x0325a8 (0xb8f6 bytes) 000206 1789584338.496147 trunc from 0x039371 to 0x028cba (0x106b6 bytes) 000207 1789584338.536585 punch from 0x020e76 to 0x021f88 (0x1113 bytes) 000209 1789584338.544381 mapread 0x0083c6 thru 0x008914 (0x54f bytes) 000210 1789584338.559346 write 0x007726 thru 0x013967 (0xc242 bytes) 000211 1789584338.584380 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28cbb): Mirror 3: 0x9ef72c4d (stale) Mirror 5: 0xed2f6844 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28cbb): Mirror 3: 0x9ef72c4d (stale) Mirror 5: 0xed2f6844 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28cbb: skipped 000212 1789584339.013146 punch from 0x015b6c to 0x01fcdb (0xa170 bytes) 000213 1789584339.020015 trunc from 0x028cbb to 0x006a23 (0x22297 bytes) 000214 1789584339.068185 write 0x0118f6 thru 0x01fef5 (0xe600 bytes) 000215 1789584339.081790 write 0x02e3df thru 0x038948 (0xa56a bytes) 000216 1789584339.085632 trunc from 0x038949 to 0x0210fe (0x1784a bytes) 000217 1789584339.133618 write 0x0140db thru 0x01d455 (0x937b bytes) 000219 1789584339.143600 read 0x00b59d thru 0x017acf (0xc533 bytes) 000220 1789584339.179039 trunc from 0x0103fb to 0x013e5a (0x3a60 bytes) 000221 1789584339.211547 write 0x0324cb thru 0x03e99e (0xc4d4 bytes) 000222 1789584339.221414 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e99f): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e99f): Mirror 3: 0xb406264 (stale) Mirror 5: 0x85a5e09d (stale) CRC-32 checksum value for chunk [0, 0x3e99f): Mirror 1: 0x9d4eb799 Mirror 6: 0x9d4eb799 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e99f: skipped 000224 1789584340.476813 write 0x026987 thru 0x02ef11 (0x858b bytes) 000225 1789584340.614914 punch from 0x005b78 to 0x006cfb (0x1184 bytes) 000226 1789584340.627006 fallocate 0x033d69 thru 0x03cdca (0x9062 bytes) 000227 1789584340.637376 read 0x03a8b1 thru 0x03e99e (0x40ee bytes) 000228 1789584340.663883 mapread 0x02e039 thru 0x036f69 (0x8f31 bytes) 000229 1789584340.722843 fallocate 0x034213 thru 0x03ffff (0xbded bytes) 000230 1789584340.737419 mapread 0x0388d5 thru 0x03ffff (0x772b bytes) 000231 1789584340.752160 write 0x03fefa thru 0x03ffff (0x106 bytes) 000232 1789584340.753569 punch from 0x03232c to 0x035425 (0x30fa bytes) 000233 1789584340.761515 fallocate 0x02ff3c thru 0x03b1a6 (0xb26b bytes) 000234 1789584340.767013 mapread 0x000623 thru 0x00e910 (0xe2ee bytes) 000235 1789584340.777427 trunc from 0x040000 to 0x0379d6 (0x8629 bytes) 000236 1789584340.828165 read 0x00b5f7 thru 0x014cd4 (0x96de bytes) 000237 1789584340.837056 punch from 0x015444 to 0x01d7fc (0x83b9 bytes) 000238 1789584340.846915 MIRROR_RESYNC 4 mirrors 000240 1789584342.529056 read 0x01c7b7 thru 0x02678b (0x9fd5 bytes) 000241 1789584342.539485 read 0x017844 thru 0x01a84a (0x3007 bytes) 000242 1789584342.557451 read 0x01efa3 thru 0x01f970 (0x9ce bytes) 000243 1789584342.557779 write 0x0190b3 thru 0x026e81 (0xddcf bytes) 000244 1789584342.605458 punch from 0x006dee to 0x0145d7 (0xd7ea bytes) 000245 1789584342.611213 mapread 0x03204f thru 0x033248 (0x11fa bytes) 000246 1789584342.612182 mapread 0x016703 thru 0x025949 (0xf247 bytes) 000247 1789584342.613820 trunc from 0x0379d7 to 0x00c5bc (0x2b41a bytes) 000248 1789584342.635825 punch from 0x0081d1 to 0x00c5bc (0x43ec bytes) 000250 1789584342.655596 write 0x03933c thru 0x03c610 (0x32d5 bytes) 000251 1789584342.656840 write 0x038277 thru 0x03ffff (0x7d89 bytes) 000252 1789584342.666564 punch from 0x03e600 to 0x03ffff (0x1a00 bytes) 000255 1789584342.698686 fallocate 0x036603 thru 0x03b54d (0x4f4b bytes) 000256 1789584342.708320 write 0x00703f thru 0x011d1b (0xacdd bytes) 000257 1789584342.731408 punch from 0x009bfa to 0x0177b0 (0xdbb7 bytes) 000258 1789584342.766943 write 0x004db0 thru 0x014c69 (0xfeba bytes) 000260 1789584342.790362 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xebb35636 (stale) Mirror 6: 0xebb35636 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xebb35636 (stale) Mirror 6: 0xebb35636 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000261 1789584343.238880 mapread 0x03a727 thru 0x03ffff (0x58d9 bytes) 000262 1789584343.267687 mapwrite 0x015cbd thru 0x01d0b9 (0x73fd bytes) 000263 1789584343.318840 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xebb35636 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xebb35636 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000265 1789584343.661346 trunc from 0x040000 to 0x0135db (0x2ca24 bytes) 000266 1789584343.701072 fallocate 0x02d816 thru 0x033d6d (0x6558 bytes) 000270 1789584343.716495 trunc from 0x004d3c to 0x00b2f0 (0x65b5 bytes) 000271 1789584343.722758 read 0x0029b6 thru 0x0096f8 (0x6d43 bytes) 000272 1789584343.731424 punch from 0x00fd68 to 0x0135db (0x3874 bytes) 000273 1789584343.742384 MIRROR_RESYNC 2 mirrors 000274 1789584344.424339 fallocate 0x02cfa6 thru 0x02ede3 (0x1e3e bytes) 000275 1789584344.467369 mapwrite 0x02b249 thru 0x035331 (0xa0e9 bytes) 000276 1789584344.545816 mapwrite 0x000605 thru 0x00c8d8 (0xc2d4 bytes) 000277 1789584344.583909 punch from 0x02c0ec to 0x035331 (0x9246 bytes) 000278 1789584344.590684 punch from 0x0213fa to 0x024d07 (0x390e bytes) 000279 1789584344.599308 write 0x028016 thru 0x031faa (0x9f95 bytes) 000280 1789584344.605760 trunc from 0x035332 to 0x03d74e (0x841d bytes) 000281 1789584344.623935 read 0x0192f4 thru 0x01d193 (0x3ea0 bytes) 000282 1789584344.635415 trunc from 0x015397 to 0x01fa4f (0xa6b9 bytes) 000283 1789584344.646188 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000285 1789584344.878937 trunc from 0x03d74f to 0x03af23 (0x282b bytes) 000286 1789584344.913237 punch from 0x01f87c to 0x02ca6a (0xd1ef bytes) 000287 1789584344.922007 trunc from 0x03af24 to 0x02ae0b (0x10118 bytes) 000290 1789584344.950939 read 0x01b294 thru 0x020c91 (0x59fe bytes) 000292 1789584344.962899 mapread 0x00ac2f thru 0x012ac4 (0x7e96 bytes) 000293 1789584344.994331 mapwrite 0x025694 thru 0x02abb6 (0x5523 bytes) 000294 1789584345.050624 trunc from 0x02ae0c to 0x01e781 (0xc68a bytes) 000295 1789584345.092014 trunc from 0x01e782 to 0x00e5dc (0x101a5 bytes) 000296 1789584345.137704 write 0x02ee8d thru 0x034900 (0x5a74 bytes) 000297 1789584345.145671 read 0x01c3af thru 0x01ff68 (0x3bba bytes) 000298 1789584345.170697 read 0x0115cd thru 0x012544 (0xf78 bytes) 000299 1789584345.172077 write 0x00a3f5 thru 0x015bb4 (0xb7c0 bytes) 000300 1789584345.174998 punch from 0x02f474 to 0x034900 (0x548d bytes) 000301 1789584345.197193 MIRROR_EXTEND to 2 mirrors 000302 1789584345.801286 fallocate 0x019f07 thru 0x02708a (0xd184 bytes) 000303 1789584345.824516 fallocate 0x03bbf2 thru 0x03ffff (0x440e bytes) 000304 1789584345.828966 trunc from 0x005d9d to 0x00db2b (0x7d8f bytes) 000305 1789584345.846966 mapread 0x0001f4 thru 0x002a60 (0x286d bytes) 000306 1789584345.885213 mapread 0x02792c thru 0x030246 (0x891b bytes) 000307 1789584345.937455 trunc from 0x034901 to 0x0144fd (0x20403 bytes) 000308 1789584346.056008 fallocate 0x002ccc thru 0x00e47f (0xb7b4 bytes) 000309 1789584346.062387 fallocate 0x0268db thru 0x026ef6 (0x61c bytes) 000310 1789584346.073746 mapwrite 0x014657 thru 0x01f1a2 (0xab4c bytes) 000311 1789584346.164492 trunc from 0x01f1a3 to 0x03dd44 (0x1eba2 bytes) 000312 1789584346.203700 mapwrite 0x03e7ea thru 0x03ffff (0x1816 bytes) 000313 1789584346.270485 mapwrite 0x0112e5 thru 0x0180aa (0x6dc6 bytes) 000315 1789584346.318993 mapread 0x02513d thru 0x025854 (0x718 bytes) 000317 1789584346.349233 trunc from 0x040000 to 0x028368 (0x17c97 bytes) 000318 1789584346.387532 trunc from 0x00c9f2 to 0x0135aa (0x6bb9 bytes) 000319 1789584346.406661 mapwrite 0x021547 thru 0x0309ff (0xf4b9 bytes) 000320 1789584346.479250 punch from 0x01566f to 0x01fb3c (0xa4ce bytes) 000321 1789584346.487053 punch from 0x02c92f to 0x0309ff (0x40d1 bytes) 000322 1789584346.494345 trunc from 0x030a00 to 0x03102b (0x62c bytes) 000323 1789584346.515277 punch from 0x007557 to 0x015fa3 (0xea4d bytes) 000325 1789584346.523532 write 0x029f73 thru 0x0357eb (0xb879 bytes) 000326 1789584346.526402 mapwrite 0x0245dd thru 0x0285a1 (0x3fc5 bytes) 000327 1789584346.552141 mapwrite 0x012611 thru 0x022025 (0xfa15 bytes) 000328 1789584346.657202 mapwrite 0x015ce0 thru 0x0202d8 (0xa5f9 bytes) 000330 1789584346.704962 punch from 0x00bfa3 to 0x00f7e5 (0x3843 bytes) 000331 1789584346.718925 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000333 1789584346.912129 mapwrite 0x00960b thru 0x01886c (0xf262 bytes) 000334 1789584347.033801 trunc from 0x027bf0 to 0x0280b2 (0x4c3 bytes) 000335 1789584347.044422 fallocate 0x01cb91 thru 0x0234df (0x694f bytes) 000336 1789584347.057023 punch from 0x0306a6 to 0x031926 (0x1281 bytes) 000337 1789584347.063706 mapread 0x0046dd thru 0x008020 (0x3944 bytes) 000338 1789584347.066287 trunc from 0x0357ec to 0x030fdc (0x480f bytes) 000339 1789584347.088770 write 0x035db4 thru 0x03ffff (0xa24c bytes) 000341 1789584347.092907 punch from 0x0369ce to 0x03d1f7 (0x682a bytes) 000342 1789584347.114576 write 0x00b733 thru 0x01811b (0xc9e9 bytes) 000343 1789584347.117595 trunc from 0x040000 to 0x006e93 (0x3916c bytes) 000344 1789584347.177184 fallocate 0x01e580 thru 0x0288b6 (0xa337 bytes) 000345 1789584347.187343 write 0x004863 thru 0x0053f1 (0xb8f bytes) 000346 1789584347.191774 fallocate 0x00808b thru 0x00ff66 (0x7edc bytes) 000347 1789584347.199611 write 0x004fab thru 0x004ff0 (0x46 bytes) 000348 1789584347.204393 mapwrite 0x0297dd thru 0x036544 (0xcd68 bytes) 000350 1789584347.298930 MIRROR_EXTEND to 2 mirrors 000352 1789584348.010638 write 0x0289b6 thru 0x02fc75 (0x72c0 bytes) 000353 1789584348.070117 trunc from 0x0262d9 to 0x02b4e2 (0x520a bytes) 000354 1789584348.088839 trunc from 0x036545 to 0x0080d2 (0x2e472 bytes) 000355 1789584348.144068 write 0x03d67b thru 0x03ffff (0x2985 bytes) 000356 1789584348.149992 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000357 1789584348.340559 fallocate 0x020a56 thru 0x022994 (0x1f3f bytes) 000358 1789584348.351623 fallocate 0x03ef0d thru 0x03ffff (0x10f3 bytes) 000359 1789584348.361053 mapread 0x000779 thru 0x006dc5 (0x664d bytes) 000360 1789584348.402778 trunc from 0x040000 to 0x010744 (0x2f8bb bytes) 000361 1789584348.440857 fallocate 0x00bbe6 thru 0x01485d (0x8c78 bytes) 000362 1789584348.448955 trunc from 0x010745 to 0x033ad7 (0x23393 bytes) 000363 1789584348.475727 punch from 0x00ae75 to 0x018d38 (0xdec4 bytes) 000364 1789584348.481952 fallocate 0x016e16 thru 0x022a41 (0xbc2c bytes) 000365 1789584348.488052 read 0x019015 thru 0x020f9a (0x7f86 bytes) 000366 1789584348.500640 trunc from 0x00b581 to 0x010421 (0x4ea1 bytes) 000368 1789584348.511359 trunc from 0x033ad8 to 0x03ea4a (0xaf73 bytes) 000370 1789584348.533794 read 0x0289f9 thru 0x031a2f (0x9037 bytes) 000372 1789584348.568757 trunc from 0x00e3eb to 0x010510 (0x2126 bytes) 000373 1789584348.573584 fallocate 0x002179 thru 0x008095 (0x5f1d bytes) 000374 1789584348.579883 write 0x00a669 thru 0x00e455 (0x3ded bytes) 000375 1789584348.585679 trunc from 0x03dfa9 to 0x03ea4a (0xaa2 bytes) 000377 1789584348.596423 trunc from 0x02c9b4 to 0x02cd77 (0x3c4 bytes) 000378 1789584348.602963 fallocate 0x00836a thru 0x010aab (0x8742 bytes) 000379 1789584348.608810 mapread 0x00654f thru 0x0106a5 (0xa157 bytes) 000382 1789584348.623062 read 0x03b2b1 thru 0x03ea4a (0x379a bytes) 000383 1789584348.633631 read 0x0301c5 thru 0x03ea4a (0xe886 bytes) 000385 1789584348.635326 read 0x031f89 thru 0x0375d5 (0x564d bytes) 000386 1789584348.636487 write 0x018290 thru 0x01fc75 (0x79e6 bytes) 000387 1789584348.637747 write 0x027ef3 thru 0x02f88c (0x799a bytes) 000388 1789584348.638894 mapwrite 0x009e65 thru 0x015446 (0xb5e2 bytes) 000389 1789584348.670073 write 0x006188 thru 0x014642 (0xe4bb bytes) 000390 1789584348.674408 mapread 0x013aec thru 0x01c142 (0x8657 bytes) 000391 1789584348.676277 read 0x00f145 thru 0x01a899 (0xb755 bytes) 000393 1789584348.683907 MIRROR_EXTEND to 2 mirrors 000394 1789584349.503477 write 0x006a25 thru 0x008194 (0x1770 bytes) 000395 1789584349.531511 mapwrite 0x00c05d thru 0x00d41a (0x13be bytes) 000396 1789584349.562070 trunc from 0x012b74 to 0x0183c8 (0x5855 bytes) 000397 1789584349.567047 mapwrite 0x035ccc thru 0x03ffff (0xa334 bytes) 000398 1789584349.640447 write 0x0161c9 thru 0x0182c9 (0x2101 bytes) 000399 1789584349.649935 write 0x006faa thru 0x00eded (0x7e44 bytes) 000400 1789584349.656389 punch from 0x01d642 to 0x02b990 (0xe34f bytes) 000401 1789584349.662066 write 0x03a661 thru 0x03ffff (0x599f bytes) 000402 1789584349.664454 trunc from 0x040000 to 0x023087 (0x1cf78 bytes) 000403 1789584349.683956 mapwrite 0x004d02 thru 0x01481e (0xfb1d bytes) 000404 1789584349.716591 mapwrite 0x03ec85 thru 0x03ffff (0x137b bytes) 000405 1789584349.752324 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000406 1789584349.884531 fallocate 0x02dce2 thru 0x02f432 (0x1751 bytes) 000407 1789584349.889947 fallocate 0x025cbf thru 0x02a4a3 (0x47e5 bytes) 000408 1789584349.895434 mapread 0x0184fa thru 0x01c6aa (0x41b1 bytes) 000409 1789584349.917993 trunc from 0x040000 to 0x00e95f (0x316a0 bytes) 000410 1789584349.941157 trunc from 0x00e960 to 0x02157f (0x12c20 bytes) 000411 1789584349.963505 fallocate 0x0063d3 thru 0x008046 (0x1c74 bytes) 000412 1789584349.970527 mapwrite 0x0265ff thru 0x032167 (0xbb69 bytes) 000413 1789584350.022037 write 0x00eba6 thru 0x01c03e (0xd499 bytes) 000414 1789584350.032577 mapread 0x017654 thru 0x01f8d3 (0x8280 bytes) 000415 1789584350.038808 mapwrite 0x003097 thru 0x005e0a (0x2d74 bytes) 000416 1789584350.060800 trunc from 0x032168 to 0x038469 (0x6302 bytes) 000417 1789584350.075656 mapread 0x00d1d6 thru 0x0181bd (0xafe8 bytes) 000418 1789584350.076711 read 0x01dc72 thru 0x02d96e (0xfcfd bytes) 000419 1789584350.077405 fallocate 0x03e0b0 thru 0x03ffff (0x1f50 bytes) 000420 1789584350.086634 fallocate 0x0260f3 thru 0x0291ae (0x30bc bytes) 000422 1789584350.102238 write 0x0055c8 thru 0x0089f8 (0x3431 bytes) 000424 1789584350.116604 fallocate 0x033f38 thru 0x03ffff (0xc0c8 bytes) 000427 1789584350.135901 write 0x014817 thru 0x022a71 (0xe25b bytes) 000428 1789584350.139846 trunc from 0x040000 to 0x030697 (0xf968 bytes) 000429 1789584350.171205 trunc from 0x030698 to 0x01813b (0x1855c bytes) 000430 1789584350.196780 mapread 0x006b05 thru 0x00de44 (0x7340 bytes) 000431 1789584350.200684 punch from 0x00dcb5 to 0x01602b (0x8377 bytes) 000433 1789584350.224524 trunc from 0x016552 to 0x01813b (0x1bea bytes) 000436 1789584350.234747 mapread 0x002ccb thru 0x00b5b0 (0x88e6 bytes) 000437 1789584350.236366 read 0x002399 thru 0x00ee0c (0xca74 bytes) 000438 1789584350.246197 read 0x006a06 thru 0x00ac59 (0x4254 bytes) 000440 1789584350.247787 write 0x0238c4 thru 0x02a25a (0x6997 bytes) 000441 1789584350.249764 mapread 0x021d9a thru 0x02a25a (0x84c1 bytes) 000445 1789584350.265356 trunc from 0x02a25b to 0x030f87 (0x6d2d bytes) 000446 1789584350.282338 fallocate 0x01b665 thru 0x0205e1 (0x4f7d bytes) 000447 1789584350.288483 mapwrite 0x03e3bf thru 0x03ffff (0x1c41 bytes) 000449 1789584350.349867 mapwrite 0x0316ab thru 0x033bbd (0x2513 bytes) 000451 1789584350.396824 punch from 0x00d3b1 to 0x011032 (0x3c82 bytes) 000452 1789584350.402631 fallocate 0x007bdd thru 0x00fdee (0x8212 bytes) 000454 1789584350.411350 mapwrite 0x00860d thru 0x011ba7 (0x959b bytes) 000455 1789584350.458049 trunc from 0x009aae to 0x00c88f (0x2de2 bytes) 000456 1789584350.464063 trunc from 0x021538 to 0x02d5cf (0xc098 bytes) 000457 1789584350.481812 MIRROR_EXTEND to 2 mirrors 000458 1789584351.497733 MIRROR_RESYNC 2 mirrors 000460 1789584351.973680 punch from 0x01eae9 to 0x026212 (0x772a bytes) 000461 1789584352.008274 punch from 0x036916 to 0x03f938 (0x9023 bytes) 000462 1789584352.016751 punch from 0x029d3d to 0x038af2 (0xedb6 bytes) 000463 1789584352.030022 fallocate 0x016959 thru 0x01f6a5 (0x8d4d bytes) 000464 1789584352.042775 punch from 0x0263e7 to 0x0313f9 (0xb013 bytes) 000465 1789584352.048406 read 0x02cedd thru 0x037f36 (0xb05a bytes) 000467 1789584352.056368 mapwrite 0x02815d thru 0x02e555 (0x63f9 bytes) 000468 1789584352.099021 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xeb348d55 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x439ea38f Mirror 3: 0x439ea38f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000469 1789584353.278492 mapread 0x0167ae thru 0x0184bb (0x1d0e bytes) 000470 1789584353.313934 trunc from 0x004bce to 0x00a88c (0x5cbf bytes) 000471 1789584353.352935 fallocate 0x002c3c thru 0x00d7a5 (0xab6a bytes) 000472 1789584353.360890 mapread 0x018f7b thru 0x025e49 (0xcecf bytes) 000473 1789584353.381782 fallocate 0x019163 thru 0x027002 (0xdea0 bytes) 000474 1789584353.389844 fallocate 0x00c0e2 thru 0x01640a (0xa329 bytes) 000475 1789584353.397038 trunc from 0x040000 to 0x026755 (0x198aa bytes) 000476 1789584353.461540 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26756): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26756): Mirror 2: 0x7edc9458 (stale) Mirror 3: 0x7edc9458 (stale) CRC-32 checksum value for chunk [0, 0x26756): Mirror 1: 0x15837911 Mirror 4: 0x15837911 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26756: skipped 000478 1789584354.816172 write 0x00cdf2 thru 0x0170ae (0xa2bd bytes) 000480 1789584354.898388 trunc from 0x026756 to 0x017838 (0xef1d bytes) 000481 1789584354.947016 mapwrite 0x030d10 thru 0x037c1a (0x6f0b bytes) 000482 1789584355.109285 trunc from 0x01fcc7 to 0x02b97d (0xbcb7 bytes) 000483 1789584355.117942 mapwrite 0x029d10 thru 0x02aecb (0x11bc bytes) 000484 1789584355.200995 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37c1b): Mirror 2: 0x75af86c1 (stale) Mirror 3: 0x27827ffa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c1b): Mirror 2: 0x75af86c1 (stale) Mirror 3: 0x27827ffa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c1b: skipped 000485 1789584355.821497 mapread 0x00eeb6 thru 0x019762 (0xa8ad bytes) 000486 1789584355.868019 fallocate 0x00b762 thru 0x012af4 (0x7393 bytes) 000487 1789584355.872981 write 0x00164d thru 0x003917 (0x22cb bytes) 000488 1789584355.888989 mapwrite 0x03aefa thru 0x03ffff (0x5106 bytes) 000489 1789584355.953433 write 0x03ad45 thru 0x03ffff (0x52bb bytes) 000490 1789584355.954949 punch from 0x0131e0 to 0x01fcb9 (0xcada bytes) 000491 1789584355.960911 read 0x01fdb9 thru 0x0206ca (0x912 bytes) 000492 1789584355.965662 trunc from 0x001336 to 0x00b6be (0xa389 bytes) 000493 1789584355.970226 punch from 0x011853 to 0x0164f9 (0x4ca7 bytes) 000494 1789584355.974936 trunc from 0x03a233 to 0x03ffff (0x5dcd bytes) 000495 1789584355.991857 trunc from 0x013147 to 0x017526 (0x43e0 bytes) 000496 1789584356.000876 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xeb348d55 (stale) Mirror 3: 0x439ea38f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xed9ab476 Mirror 4: 0xed9ab476 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000498 1789584357.302426 fallocate 0x009d8c thru 0x011247 (0x74bc bytes) 000499 1789584357.386881 trunc from 0x00090e to 0x00920e (0x8901 bytes) 000500 1789584357.399033 write 0x031793 thru 0x0323c2 (0xc30 bytes) 000501 1789584357.449964 read 0x038319 thru 0x03ffff (0x7ce7 bytes) 000502 1789584357.462998 trunc from 0x009af9 to 0x00e3ca (0x48d2 bytes) 000503 1789584357.480308 fallocate 0x024b3d thru 0x026f2c (0x23f0 bytes) 000504 1789584357.490872 fallocate 0x03cdbb thru 0x03ffff (0x3245 bytes) 000505 1789584357.496433 fallocate 0x01a0ec thru 0x029462 (0xf377 bytes) 000506 1789584357.502939 fallocate 0x018cfc thru 0x019d5e (0x1063 bytes) 000507 1789584357.512887 MIRROR_RESYNC 4 mirrors 000509 1789584359.094818 punch from 0x009359 to 0x00d829 (0x44d1 bytes) 000510 1789584359.128263 punch from 0x03678c to 0x03a9c6 (0x423b bytes) 000511 1789584359.135390 punch from 0x0168a3 to 0x023b35 (0xd293 bytes) 000512 1789584359.142167 fallocate 0x01757c thru 0x01ddd3 (0x6858 bytes) 000513 1789584359.147406 punch from 0x00574c to 0x00801a (0x28cf bytes) 000515 1789584359.152864 mapread 0x0315b9 thru 0x03ce11 (0xb859 bytes) 000516 1789584359.178227 trunc from 0x02526e to 0x02794c (0x26df bytes) 000517 1789584359.184397 trunc from 0x008fcd to 0x00f311 (0x6345 bytes) 000518 1789584359.190362 trunc from 0x040000 to 0x0188fe (0x27701 bytes) 000519 1789584359.219629 punch from 0x002525 to 0x00f38a (0xce66 bytes) 000520 1789584359.225564 mapread 0x015f91 thru 0x0188fe (0x296e bytes) 000521 1789584359.244366 trunc from 0x0188ff to 0x016c03 (0x1cfb bytes) 000522 1789584359.273306 write 0x005018 thru 0x00d69e (0x8687 bytes) 000523 1789584359.285327 fallocate 0x039004 thru 0x03ffff (0x6ffc bytes) 000525 1789584359.291742 punch from 0x00ef28 to 0x0115eb (0x26c4 bytes) 000526 1789584359.301293 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x16c04): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16c04): Mirror 2: 0x45215fe (stale) Mirror 3: 0x45215fe (stale) Mirror 4: 0x45215fe (stale) CRC-32 checksum value for chunk [0, 0x16c04): Mirror 1: 0xab313f35 Mirror 5: 0xab313f35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16c04: skipped 000527 1789584360.231725 fallocate 0x01d33d thru 0x01f4a5 (0x2169 bytes) 000529 1789584360.304872 trunc from 0x01f4a6 to 0x03e87c (0x1f3d7 bytes) 000531 1789584360.352815 trunc from 0x01fec3 to 0x0209de (0xb1c bytes) 000532 1789584360.375856 mapwrite 0x0191f8 thru 0x01b299 (0x20a2 bytes) 000533 1789584360.445443 write 0x0391f2 thru 0x03c7c8 (0x35d7 bytes) 000534 1789584360.463869 mapwrite 0x0355d9 thru 0x03a0e1 (0x4b09 bytes) 000535 1789584360.500769 punch from 0x02153e to 0x0302f0 (0xedb3 bytes) 000536 1789584360.506867 trunc from 0x023360 to 0x023aaf (0x750 bytes) 000537 1789584360.512342 write 0x030e38 thru 0x03f318 (0xe4e1 bytes) 000538 1789584360.520877 mapread 0x01533d thru 0x01efbb (0x9c7f bytes) 000539 1789584360.523554 mapread 0x02612d thru 0x033d0e (0xdbe2 bytes) 000541 1789584360.541972 fallocate 0x01ff68 thru 0x02c6c8 (0xc761 bytes) 000542 1789584360.553537 read 0x015f0f thru 0x01ee84 (0x8f76 bytes) 000543 1789584360.555374 read 0x00443a thru 0x004a9e (0x665 bytes) 000544 1789584360.573637 trunc from 0x03f319 to 0x0235d4 (0x1bd44 bytes) 000545 1789584360.593752 trunc from 0x0235d5 to 0x01b7d1 (0x7e03 bytes) 000546 1789584360.619300 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b7d2): Mirror 3: 0xccc1b93d (stale) Mirror 4: 0xccc1b93d (stale) Mirror 5: 0xeaff6bd9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b7d2): Mirror 3: 0xccc1b93d (stale) Mirror 4: 0xccc1b93d (stale) Mirror 5: 0xeaff6bd9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b7d2: skipped 000547 1789584361.262478 fallocate 0x0081bc thru 0x01161c (0x9461 bytes) 000548 1789584361.290956 mapwrite 0x02cce2 thru 0x02ff96 (0x32b5 bytes) 000549 1789584361.438323 write 0x03d75c thru 0x03ffff (0x28a4 bytes) 000550 1789584361.444798 mapread 0x0349df thru 0x034d4b (0x36d bytes) 000552 1789584361.468811 mapread 0x00eb6c thru 0x01bf41 (0xd3d6 bytes) 000553 1789584361.504811 mapwrite 0x00cd05 thru 0x01632c (0x9628 bytes) 000554 1789584361.576498 mapread 0x03a392 thru 0x03ffff (0x5c6e bytes) 000555 1789584361.578013 fallocate 0x028682 thru 0x02a417 (0x1d96 bytes) 000556 1789584361.601244 trunc from 0x040000 to 0x0021ca (0x3de35 bytes) 000557 1789584361.618355 read 0x000380 thru 0x0021ca (0x1e4b bytes) 000558 1789584361.625202 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21cb): Mirror 3: 0xe83bce60 (stale) Mirror 5: 0xe83bce60 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21cb): Mirror 3: 0xe83bce60 (stale) Mirror 5: 0xe83bce60 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21cb: skipped 000560 1789584361.928683 mapwrite 0x019cbf thru 0x01f0d9 (0x541b bytes) 000561 1789584361.976167 mapread 0x01d54f thru 0x01f0d9 (0x1b8b bytes) 000562 1789584361.976858 trunc from 0x01f0da to 0x03e274 (0x1f19b bytes) 000563 1789584362.002687 MIRROR_RESYNC 3 mirrors 000564 1789584363.200765 write 0x01be24 thru 0x021880 (0x5a5d bytes) 000565 1789584363.251929 write 0x0209c0 thru 0x020c91 (0x2d2 bytes) 000566 1789584363.252933 trunc from 0x00671a to 0x00b04b (0x4932 bytes) 000567 1789584363.260916 read 0x0375b9 thru 0x03e274 (0x6cbc bytes) 000568 1789584363.270778 fallocate 0x0275ac thru 0x03519b (0xdbf0 bytes) 000569 1789584363.284659 punch from 0x017ba6 to 0x02774a (0xfba5 bytes) 000570 1789584363.309162 mapwrite 0x01218f thru 0x01d974 (0xb7e6 bytes) 000571 1789584363.372240 trunc from 0x03701b to 0x03e274 (0x725a bytes) 000572 1789584363.384922 fallocate 0x010b28 thru 0x01112b (0x604 bytes) 000573 1789584363.392825 mapwrite 0x00759e thru 0x016f89 (0xf9ec bytes) 000575 1789584363.433052 trunc from 0x01a891 to 0x01e515 (0x3c85 bytes) 000576 1789584363.438289 read 0x00da1e thru 0x01967b (0xbc5e bytes) 000577 1789584363.439041 punch from 0x02ad0e to 0x035f95 (0xb288 bytes) 000578 1789584363.443969 fallocate 0x0081e8 thru 0x00a29d (0x20b6 bytes) 000579 1789584363.449835 read 0x00c738 thru 0x016605 (0x9ece bytes) 000580 1789584363.450825 mapread 0x01263a thru 0x01c5c0 (0x9f87 bytes) 000581 1789584363.463304 trunc from 0x03e275 to 0x0198ba (0x249ba bytes) 000582 1789584363.496879 trunc from 0x0198bb to 0x02280b (0x8f51 bytes) 000583 1789584363.516242 punch from 0x0106ad to 0x014ecb (0x481f bytes) 000584 1789584363.523787 fallocate 0x0155f0 thru 0x022f85 (0xd996 bytes) 000585 1789584363.529718 trunc from 0x007a73 to 0x007e25 (0x3b3 bytes) 000586 1789584363.540686 fallocate 0x03d702 thru 0x03ffff (0x28fe bytes) 000587 1789584363.547397 fallocate 0x0232d3 thru 0x028580 (0x52ae bytes) 000588 1789584363.556652 punch from 0x01d513 to 0x01eb94 (0x1682 bytes) 000589 1789584363.561729 punch from 0x008f59 to 0x00cd44 (0x3dec bytes) 000590 1789584363.568127 read 0x000d3e thru 0x00f575 (0xe838 bytes) 000591 1789584363.662482 punch from 0x000707 to 0x005ca6 (0x55a0 bytes) 000592 1789584363.676691 write 0x022d3d thru 0x02c2a2 (0x9566 bytes) 000593 1789584363.692239 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c2a3): Mirror 5: 0x2bb17eac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c2a3): Mirror 5: 0x2bb17eac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c2a3: skipped 000594 1789584364.343631 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000595 1789584364.668693 fallocate 0x01f50c thru 0x021156 (0x1c4b bytes) 000596 1789584364.679692 read 0x006e23 thru 0x013654 (0xc832 bytes) 000597 1789584364.702609 write 0x03f819 thru 0x03ffff (0x7e7 bytes) 000598 1789584364.729009 mapwrite 0x0172df thru 0x02236b (0xb08d bytes) 000599 1789584364.835647 read 0x004907 thru 0x00be2f (0x7529 bytes) 000600 1789584364.852996 mapread 0x0172b1 thru 0x020a4a (0x979a bytes) 000601 1789584364.853988 trunc from 0x00880b to 0x00e318 (0x5b0e bytes) 000603 1789584364.868775 trunc from 0x040000 to 0x016a6b (0x29594 bytes) 000604 1789584364.903452 fallocate 0x02a2c3 thru 0x030364 (0x60a2 bytes) 000605 1789584364.919898 punch from 0x005e7c to 0x01211f (0xc2a4 bytes) 000606 1789584364.933001 write 0x01ac71 thru 0x024988 (0x9d18 bytes) 000607 1789584364.940767 trunc from 0x01dffe to 0x020ed4 (0x2ed7 bytes) 000608 1789584364.988678 write 0x024de2 thru 0x02c978 (0x7b97 bytes) 000610 1789584364.995911 mapwrite 0x00eee6 thru 0x0108cd (0x19e8 bytes) 000611 1789584365.088500 MIRROR_EXTEND to 2 mirrors 000612 1789584366.049069 write 0x025b78 thru 0x02fff2 (0xa47b bytes) 000613 1789584366.129303 mapwrite 0x00c3bb thru 0x00ffc6 (0x3c0c bytes) 000614 1789584366.187422 write 0x02fe1d thru 0x0328d5 (0x2ab9 bytes) 000615 1789584366.188751 trunc from 0x01c6fd to 0x02ace3 (0xe5e7 bytes) 000618 1789584366.198912 read 0x029f91 thru 0x0328d5 (0x8945 bytes) 000619 1789584366.206196 mapwrite 0x00bece thru 0x00d20c (0x133f bytes) 000620 1789584366.233284 MIRROR_RESYNC 2 mirrors 000621 1789584367.222797 read 0x01d97a thru 0x01ff5e (0x25e5 bytes) 000622 1789584367.228439 read 0x029b1a thru 0x0328d5 (0x8dbc bytes) 000623 1789584367.256685 mapwrite 0x014449 thru 0x01b6e9 (0x72a1 bytes) 000624 1789584367.493550 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x328d6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x328d6): Mirror 2: 0xab224f34 (stale) CRC-32 checksum value for chunk [0, 0x328d6): Mirror 1: 0xfc693c17 Mirror 3: 0xfc693c17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x328d6: skipped 000626 1789584369.096468 trunc from 0x0328d6 to 0x009d2e (0x28ba7 bytes) 000627 1789584369.181640 trunc from 0x009d2f to 0x0022bb (0x7a73 bytes) 000628 1789584369.225753 mapread 0x0008a2 thru 0x0022bb (0x1a1a bytes) 000629 1789584369.255875 trunc from 0x00154b to 0x0022bb (0xd71 bytes) 000630 1789584369.270576 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22bc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22bc): Mirror 2: 0x747314f5 (stale) Mirror 3: 0x747314f5 (stale) CRC-32 checksum value for chunk [0, 0x22bc): Mirror 1: 0x747314f5 Mirror 4: 0x747314f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22bc: skipped 000631 1789584370.393004 trunc from 0x0022bc to 0x017dee (0x15b33 bytes) 000632 1789584370.509028 MIRROR_RESYNC 4 mirrors 000633 1789584371.515184 trunc from 0x017def to 0x00f583 (0x886b bytes) 000634 1789584371.562620 trunc from 0x00f584 to 0x01bb89 (0xc606 bytes) 000635 1789584371.599014 write 0x023a1a thru 0x029c3c (0x6223 bytes) 000637 1789584371.603282 mapwrite 0x00225d thru 0x006746 (0x44ea bytes) 000638 1789584371.660583 mapread 0x0180c2 thru 0x01afd1 (0x2f10 bytes) 000639 1789584371.676182 read 0x0206c1 thru 0x028af9 (0x8439 bytes) 000640 1789584371.684602 mapwrite 0x007401 thru 0x00e53c (0x713c bytes) 000642 1789584371.720674 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29c3d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c3d): Mirror 2: 0x5620da7c (stale) Mirror 3: 0x5620da7c (stale) Mirror 4: 0x5620da7c (stale) CRC-32 checksum value for chunk [0, 0x29c3d): Mirror 1: 0xde3483a7 Mirror 5: 0xde3483a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c3d: skipped 000643 1789584373.159572 mapwrite 0x007fed thru 0x015a04 (0xda18 bytes) 000644 1789584373.394979 read 0x022195 thru 0x029c3c (0x7aa8 bytes) 000646 1789584373.430793 mapwrite 0x00922b thru 0x00a6c9 (0x149f bytes) 000647 1789584373.468674 trunc from 0x029c3d to 0x01a7ad (0xf48f bytes) 000648 1789584373.565644 trunc from 0x01a7ae to 0x03d09a (0x228ed bytes) 000649 1789584373.593882 mapread 0x016360 thru 0x01682d (0x4ce bytes) 000650 1789584373.595964 mapwrite 0x029cb1 thru 0x03541d (0xb76d bytes) 000651 1789584373.691821 trunc from 0x03d09b to 0x020cee (0x1c3ac bytes) 000652 1789584373.731961 fallocate 0x003936 thru 0x011755 (0xde20 bytes) 000653 1789584373.739301 fallocate 0x00ebfe thru 0x013837 (0x4c3a bytes) 000654 1789584373.757935 punch from 0x018c92 to 0x01c4ec (0x385b bytes) 000655 1789584373.769612 trunc from 0x020cef to 0x018cb5 (0x8039 bytes) 000656 1789584373.830264 fallocate 0x027111 thru 0x02b347 (0x4237 bytes) 000657 1789584373.847281 mapread 0x00ac48 thru 0x00c814 (0x1bcd bytes) 000658 1789584373.848664 punch from 0x009b8e to 0x00bada (0x1f4d bytes) 000659 1789584373.856679 trunc from 0x018cb6 to 0x025e77 (0xd1c2 bytes) 000660 1789584373.896888 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25e78): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25e78): Mirror 2: 0x3cd1ab4c (stale) Mirror 3: 0x3cd1ab4c (stale) Mirror 4: 0x3cd1ab4c (stale) Mirror 5: 0x4dfb79ce (stale) CRC-32 checksum value for chunk [0, 0x25e78): Mirror 1: 0x554a967f Mirror 6: 0x554a967f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25e78: skipped 000661 1789584375.379526 read 0x01989d thru 0x0223ab (0x8b0f bytes) 000662 1789584375.421820 fallocate 0x00809f thru 0x00d8bd (0x581f bytes) 000663 1789584375.519208 trunc from 0x001818 to 0x00d41b (0xbc04 bytes) 000664 1789584375.546311 punch from 0x008f78 to 0x015500 (0xc589 bytes) 000665 1789584375.561415 trunc from 0x014ec7 to 0x01f824 (0xa95e bytes) 000666 1789584375.584699 punch from 0x0104d1 to 0x01447d (0x3fad bytes) 000667 1789584375.592826 fallocate 0x03789a thru 0x03ffff (0x8766 bytes) 000668 1789584375.599829 trunc from 0x040000 to 0x01944e (0x26bb1 bytes) 000670 1789584375.658289 mapread 0x00422c thru 0x0094a0 (0x5275 bytes) 000671 1789584375.715040 trunc from 0x01944f to 0x03ab06 (0x216b8 bytes) 000672 1789584375.743462 mapwrite 0x02f805 thru 0x037a0f (0x820b bytes) 000673 1789584375.804034 trunc from 0x0029d4 to 0x01096b (0xdf98 bytes) 000674 1789584375.819972 trunc from 0x018ee2 to 0x01c88f (0x39ae bytes) 000675 1789584375.828645 mapread 0x034e93 thru 0x03ab06 (0x5c74 bytes) 000676 1789584375.838018 mapread 0x013f04 thru 0x018a8a (0x4b87 bytes) 000677 1789584375.859787 trunc from 0x00469e to 0x00e595 (0x9ef8 bytes) 000678 1789584375.868177 trunc from 0x001b33 to 0x008e42 (0x7310 bytes) 000679 1789584375.875949 mapwrite 0x015bb3 thru 0x01711b (0x1569 bytes) 000680 1789584375.896734 fallocate 0x03cd1d thru 0x03ffff (0x32e3 bytes) 000681 1789584375.917747 trunc from 0x03ab07 to 0x026407 (0x146ff bytes) 000682 1789584375.956061 mapread 0x01d6ae thru 0x026407 (0x8d5a bytes) 000683 1789584375.975940 mapread 0x01666c thru 0x01f85c (0x91f1 bytes) 000684 1789584375.976941 fallocate 0x01eac4 thru 0x02e414 (0xf951 bytes) 000685 1789584375.984256 fallocate 0x0088d4 thru 0x00bdf3 (0x3520 bytes) 000686 1789584375.990197 mapwrite 0x017696 thru 0x026ad1 (0xf43c bytes) 000687 1789584376.044364 punch from 0x0011e2 to 0x001db6 (0xbd5 bytes) 000689 1789584376.066719 read 0x0040be thru 0x00fe12 (0xbd55 bytes) 000690 1789584376.088625 trunc from 0x026ad2 to 0x00f804 (0x172cd bytes) 000691 1789584376.111360 read 0x00c026 thru 0x00c1b0 (0x18b bytes) 000692 1789584376.112288 mapwrite 0x01e255 thru 0x02d13a (0xeee6 bytes) 000693 1789584376.186139 fallocate 0x025211 thru 0x0260ae (0xe9e bytes) 000694 1789584376.195076 punch from 0x018633 to 0x0266a6 (0xe074 bytes) 000695 1789584376.204049 trunc from 0x00025d to 0x0098fc (0x96a0 bytes) 000696 1789584376.213631 mapwrite 0x015686 thru 0x0226ad (0xd028 bytes) 000697 1789584376.329613 trunc from 0x02d13b to 0x02ef80 (0x1e46 bytes) 000698 1789584376.378825 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ef81): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ef81): Mirror 2: 0x4377519 (stale) Mirror 3: 0x4377519 (stale) Mirror 4: 0x4377519 (stale) Mirror 5: 0x6f3c34e9 (stale) Mirror 6: 0x8def4ae (stale) CRC-32 checksum value for chunk [0, 0x2ef81): Mirror 1: 0x1efdc176 Mirror 7: 0x1efdc176 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ef81: skipped 000700 1789584377.874632 MIRROR_RESYNC 7 mirrors 000701 1789584379.984412 mapread 0x026f43 thru 0x02ef80 (0x803e bytes) 000702 1789584380.041351 write 0x004560 thru 0x010e23 (0xc8c4 bytes) 000703 1789584380.156554 mapwrite 0x025d60 thru 0x02bebc (0x615d bytes) 000704 1789584380.203230 punch from 0x022f85 to 0x02e7ae (0xb82a bytes) 000705 1789584380.211684 mapwrite 0x0337ac thru 0x03409f (0x8f4 bytes) 000708 1789584380.278108 trunc from 0x01dd7b to 0x028c95 (0xaf1b bytes) 000709 1789584380.297204 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x340a0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340a0): Mirror 2: 0x74fdd1e5 (stale) Mirror 3: 0x74fdd1e5 (stale) Mirror 4: 0x74fdd1e5 (stale) Mirror 5: 0x74fdd1e5 (stale) Mirror 6: 0x74fdd1e5 (stale) Mirror 7: 0x74fdd1e5 (stale) CRC-32 checksum value for chunk [0, 0x340a0): Mirror 1: 0xaeacd3fc Mirror 8: 0xaeacd3fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340a0: skipped 000710 1789584381.587994 punch from 0x02335b to 0x0235e0 (0x286 bytes) 000711 1789584381.676415 read 0x0223b5 thru 0x025e75 (0x3ac1 bytes) 000712 1789584381.681473 trunc from 0x0340a0 to 0x018412 (0x1bc8d bytes) 000713 1789584381.748708 punch from 0x0109d6 to 0x018412 (0x7a3d bytes) 000714 1789584381.756982 mapwrite 0x012bfc thru 0x014765 (0x1b6a bytes) 000715 1789584381.819581 MIRROR_RESYNC 8 mirrors 000716 1789584384.336676 trunc from 0x018413 to 0x02cff1 (0x14bdf bytes) 000717 1789584384.453571 mapwrite 0x03538a thru 0x03ffff (0xac76 bytes) 000718 1789584384.588662 trunc from 0x040000 to 0x02a3ca (0x15c35 bytes) 000719 1789584384.605923 trunc from 0x01ff87 to 0x0209a4 (0xa1e bytes) 000720 1789584384.616363 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a3cb): Mirror 2: 0xea4f6ae9 (stale) Mirror 3: 0xea4f6ae9 (stale) Mirror 4: 0xea4f6ae9 (stale) Mirror 5: 0xea4f6ae9 (stale) Mirror 7: 0xea4f6ae9 (stale) Mirror 8: 0xea4f6ae9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a3cb): Mirror 2: 0xea4f6ae9 (stale) Mirror 3: 0xea4f6ae9 (stale) Mirror 4: 0xea4f6ae9 (stale) Mirror 5: 0xea4f6ae9 (stale) Mirror 7: 0xea4f6ae9 (stale) Mirror 8: 0xea4f6ae9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a3cb: skipped 000721 1789584385.040507 trunc from 0x02a3cb to 0x013272 (0x17158 bytes) 000722 1789584385.063421 read 0x00373d thru 0x008104 (0x49c8 bytes) 000723 1789584385.074872 mapread 0x003f2b thru 0x011a68 (0xdb3e bytes) 000724 1789584385.094001 trunc from 0x010bd1 to 0x013272 (0x26a2 bytes) 000725 1789584385.099902 read 0x00b462 thru 0x013272 (0x7e11 bytes) 000726 1789584385.111959 trunc from 0x00249e to 0x00ccf7 (0xa85a bytes) 000727 1789584385.119834 trunc from 0x010456 to 0x013272 (0x2e1d bytes) 000728 1789584385.125550 write 0x024b93 thru 0x02d9f9 (0x8e67 bytes) 000729 1789584385.129061 trunc from 0x01efb8 to 0x02b6b4 (0xc6fd bytes) 000730 1789584385.150210 read 0x0026e4 thru 0x00c01f (0x993c bytes) 000731 1789584385.175592 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d9fa): Mirror 2: 0xcbeb9fe3 (stale) Mirror 3: 0xcbeb9fe3 (stale) Mirror 4: 0xcbeb9fe3 (stale) Mirror 5: 0xcbeb9fe3 (stale) Mirror 7: 0xcbeb9fe3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d9fa): Mirror 2: 0xcbeb9fe3 (stale) Mirror 3: 0xcbeb9fe3 (stale) Mirror 4: 0xcbeb9fe3 (stale) Mirror 5: 0xcbeb9fe3 (stale) Mirror 7: 0xcbeb9fe3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d9fa: skipped 000732 1789584385.623068 write 0x03760c thru 0x03eb50 (0x7545 bytes) 000733 1789584385.638840 write 0x03e9cd thru 0x03ffff (0x1633 bytes) 000734 1789584385.640014 write 0x024fd2 thru 0x02ba88 (0x6ab7 bytes) 000735 1789584385.653677 write 0x02c1c9 thru 0x0322c6 (0x60fe bytes) 000736 1789584385.665840 trunc from 0x040000 to 0x027774 (0x1888b bytes) 000737 1789584385.687048 fallocate 0x0120c7 thru 0x01ad30 (0x8c6a bytes) 000738 1789584385.693583 fallocate 0x012cd4 thru 0x017037 (0x4364 bytes) 000739 1789584385.698909 mapwrite 0x01a280 thru 0x021ba0 (0x7921 bytes) 000740 1789584385.741454 mapread 0x01766a thru 0x017877 (0x20e bytes) 000741 1789584385.742579 trunc from 0x027775 to 0x015c83 (0x11af1 bytes) 000742 1789584385.764185 write 0x003f0b thru 0x00e39d (0xa493 bytes) 000743 1789584385.776034 mapread 0x004629 thru 0x00741b (0x2df3 bytes) 000744 1789584385.777433 mapwrite 0x029306 thru 0x03822b (0xef26 bytes) 000745 1789584385.857920 read 0x0268db thru 0x031221 (0xa947 bytes) 000746 1789584385.858818 punch from 0x017ce0 to 0x01be3b (0x415c bytes) 000747 1789584385.865399 mapread 0x023b3c thru 0x02505e (0x1523 bytes) 000748 1789584385.866293 mapread 0x01cbba thru 0x01f1ff (0x2646 bytes) 000749 1789584385.876436 trunc from 0x008165 to 0x011d9d (0x9c39 bytes) 000750 1789584385.886606 write 0x038e28 thru 0x03ffff (0x71d8 bytes) 000751 1789584385.889211 write 0x00dfac thru 0x010aef (0x2b44 bytes) 000752 1789584385.903869 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa3de3210 (stale) Mirror 4: 0xa3de3210 (stale) Mirror 5: 0xa3de3210 (stale) Mirror 7: 0xa3de3210 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa3de3210 (stale) Mirror 4: 0xa3de3210 (stale) Mirror 5: 0xa3de3210 (stale) Mirror 7: 0xa3de3210 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000753 1789584386.368964 trunc from 0x040000 to 0x0019e2 (0x3e61d bytes) 000754 1789584386.405178 mapwrite 0x01c805 thru 0x01fa80 (0x327c bytes) 000755 1789584386.485909 mapwrite 0x02e0d2 thru 0x03de4a (0xfd79 bytes) 000756 1789584386.541053 fallocate 0x028cb6 thru 0x02bcb7 (0x3002 bytes) 000757 1789584386.546324 trunc from 0x017b29 to 0x025c2d (0xe105 bytes) 000758 1789584386.554041 write 0x00e659 thru 0x014c14 (0x65bc bytes) 000759 1789584386.566478 mapwrite 0x034714 thru 0x0347b2 (0x9f bytes) 000760 1789584386.584958 mapwrite 0x01183a thru 0x01547d (0x3c44 bytes) 000761 1789584386.612102 trunc from 0x0383f3 to 0x039b8b (0x1799 bytes) 000762 1789584386.617379 trunc from 0x028bc0 to 0x02b893 (0x2cd4 bytes) 000763 1789584386.622624 trunc from 0x03de4b to 0x03d92e (0x51c bytes) 000764 1789584386.638692 fallocate 0x01f53b thru 0x026403 (0x6ec9 bytes) 000765 1789584386.644766 trunc from 0x01f7da to 0x021b3a (0x2361 bytes) 000766 1789584386.649985 fallocate 0x00b300 thru 0x015859 (0xa55a bytes) 000767 1789584386.659829 mapread 0x003ad2 thru 0x00b01e (0x754d bytes) 000768 1789584386.669545 read 0x0153e3 thru 0x015d8e (0x9ac bytes) 000769 1789584386.670239 write 0x006b90 thru 0x00af4f (0x43c0 bytes) 000770 1789584386.672526 read 0x0259b7 thru 0x0342d1 (0xe91b bytes) 000771 1789584386.683373 trunc from 0x00f905 to 0x010f88 (0x1684 bytes) 000772 1789584386.688158 fallocate 0x00ad9d thru 0x00ba30 (0xc94 bytes) 000773 1789584386.694301 read 0x014eeb thru 0x01a5d1 (0x56e7 bytes) 000774 1789584386.695268 read 0x006049 thru 0x006304 (0x2bc bytes) 000775 1789584386.696499 read 0x013b7f thru 0x022b97 (0xf019 bytes) 000776 1789584386.697604 mapread 0x034174 thru 0x03d92e (0x97bb bytes) 000777 1789584386.699491 mapread 0x0293c6 thru 0x030408 (0x7043 bytes) 000778 1789584386.701698 read 0x01c672 thru 0x026bd6 (0xa565 bytes) 000779 1789584386.707667 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d92f): Mirror 2: 0x5a57b249 (stale) Mirror 4: 0x5a57b249 (stale) Mirror 7: 0x5a57b249 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d92f): Mirror 2: 0x5a57b249 (stale) Mirror 4: 0x5a57b249 (stale) Mirror 7: 0x5a57b249 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d92f: skipped 000780 1789584387.082699 mapread 0x02ddd6 thru 0x03b083 (0xd2ae bytes) 000781 1789584387.113388 punch from 0x03d149 to 0x03d764 (0x61c bytes) 000782 1789584387.118602 write 0x01e9b2 thru 0x027188 (0x87d7 bytes) 000783 1789584387.131716 mapwrite 0x012312 thru 0x01737b (0x506a bytes) 000784 1789584387.180368 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d92f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d92f): Mirror 2: 0x5a57b249 (stale) Mirror 4: 0x5a57b249 (stale) Mirror 7: 0x5a57b249 (stale) CRC-32 checksum value for chunk [0, 0x3d92f): Mirror 1: 0x3fde7224 Mirror 8: 0x3fde7224 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d92f: skipped 000785 1789584388.435704 punch from 0x013cda to 0x01a4f8 (0x681f bytes) 000786 1789584388.509323 MIRROR_RESYNC 5 mirrors 000787 1789584390.835704 trunc from 0x023ff1 to 0x033eee (0xfefe bytes) 000788 1789584390.897924 punch from 0x0179b8 to 0x023461 (0xbaaa bytes) 000789 1789584390.908957 fallocate 0x0058e0 thru 0x00d222 (0x7943 bytes) 000790 1789584390.920954 trunc from 0x03d92f to 0x031d6a (0xbbc4 bytes) 000791 1789584390.974043 write 0x03e6c0 thru 0x03ffff (0x1940 bytes) 000792 1789584390.976913 read 0x03a3fc thru 0x03ffff (0x5c04 bytes) 000793 1789584390.995280 fallocate 0x02110a thru 0x025517 (0x440e bytes) 000794 1789584391.004630 mapwrite 0x03614c thru 0x03ffff (0x9eb4 bytes) 000795 1789584391.103698 trunc from 0x007645 to 0x011f07 (0xa8c3 bytes) 000796 1789584391.112047 punch from 0x005997 to 0x014266 (0xe8d0 bytes) 000797 1789584391.120484 trunc from 0x01f673 to 0x02a646 (0xafd4 bytes) 000798 1789584391.126393 fallocate 0x03cd1d thru 0x03d60b (0x8ef bytes) 000799 1789584391.135914 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x524e23cf (stale) Mirror 4: 0x524e23cf (stale) Mirror 7: 0x524e23cf (stale) Mirror 8: 0x524e23cf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1b0957a4 Mirror 9: 0x1b0957a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000800 1789584392.312809 punch from 0x00533a to 0x013413 (0xe0da bytes) 000801 1789584392.365519 write 0x0394c4 thru 0x03ffff (0x6b3c bytes) 000802 1789584392.384427 punch from 0x00382d to 0x011ffb (0xe7cf bytes) 000804 1789584392.388730 trunc from 0x040000 to 0x00f28d (0x30d72 bytes) 000805 1789584392.435504 write 0x026fd1 thru 0x02d6e4 (0x6714 bytes) 000807 1789584392.440421 read 0x018ad3 thru 0x01da29 (0x4f57 bytes) 000808 1789584392.465600 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2d6e5): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d6e5): Mirror 2: 0xc16a880c (stale) Mirror 4: 0xc16a880c (stale) Mirror 7: 0xc16a880c (stale) Mirror 8: 0xc16a880c (stale) Mirror 9: 0x1d2bb4cb (stale) CRC-32 checksum value for chunk [0, 0x2d6e5): Mirror 1: 0x16edfd4e Mirror 10: 0x16edfd4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d6e5: skipped 000810 1789584393.196779 punch from 0x020e23 to 0x025d44 (0x4f22 bytes) 000811 1789584393.251797 mapwrite 0x02be3c thru 0x0301d5 (0x439a bytes) 000812 1789584393.408923 fallocate 0x00310b thru 0x0128ab (0xf7a1 bytes) 000813 1789584393.420388 mapwrite 0x008001 thru 0x00f1ba (0x71ba bytes) 000814 1789584393.458216 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x301d6): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301d6): Mirror 2: 0x99463151 (stale) Mirror 4: 0x99463151 (stale) Mirror 7: 0x99463151 (stale) Mirror 8: 0x99463151 (stale) Mirror 9: 0x3eba0d70 (stale) Mirror 10: 0x9ce93af6 (stale) CRC-32 checksum value for chunk [0, 0x301d6): Mirror 1: 0x55052fe5 Mirror 11: 0x55052fe5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301d6: skipped 000815 1789584394.521751 mapwrite 0x03965b thru 0x03a304 (0xcaa bytes) 000816 1789584394.702619 mapwrite 0x03d16a thru 0x03ffff (0x2e96 bytes) 000817 1789584394.784483 fallocate 0x02bf95 thru 0x038aa2 (0xcb0e bytes) 000818 1789584394.793155 mapread 0x02f6e3 thru 0x035b85 (0x64a3 bytes) 000819 1789584394.823040 mapwrite 0x03091b thru 0x031dfc (0x14e2 bytes) 000820 1789584394.843623 mapwrite 0x006806 thru 0x0137f8 (0xcff3 bytes) 000821 1789584394.924941 read 0x033ac4 thru 0x038b05 (0x5042 bytes) 000822 1789584394.926494 mapread 0x022a3e thru 0x023583 (0xb46 bytes) 000823 1789584394.946062 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x524e23cf (stale) Mirror 4: 0x524e23cf (stale) Mirror 7: 0x524e23cf (stale) Mirror 8: 0x524e23cf (stale) Mirror 9: 0x1b0957a4 (stale) Mirror 10: 0xcd5b381a (stale) Mirror 11: 0xda0c8c3b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7ced14cd Mirror 12: 0x7ced14cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000824 1789584395.810800 trunc from 0x040000 to 0x01536f (0x2ac90 bytes) 000825 1789584395.884959 mapread 0x007043 thru 0x00af99 (0x3f57 bytes) 000827 1789584395.912828 read 0x007c85 thru 0x011462 (0x97de bytes) 000828 1789584395.915042 mapread 0x014cf8 thru 0x01536f (0x678 bytes) 000829 1789584395.929350 trunc from 0x00a63e to 0x00d8ff (0x32c2 bytes) 000830 1789584395.941012 write 0x02e006 thru 0x030c56 (0x2c51 bytes) 000831 1789584395.942288 fallocate 0x0031b2 thru 0x005a93 (0x28e2 bytes) 000832 1789584395.953329 fallocate 0x03ff3b thru 0x03ffff (0xc5 bytes) 000833 1789584395.962327 fallocate 0x021875 thru 0x030a12 (0xf19e bytes) 000834 1789584395.972877 trunc from 0x01b5cc to 0x029ff6 (0xea2b bytes) 000835 1789584395.986663 fallocate 0x028a3d thru 0x02c969 (0x3f2d bytes) 000836 1789584395.997572 write 0x02ee15 thru 0x037b18 (0x8d04 bytes) 000837 1789584396.009504 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37b19): Mirror 2: 0x6a8d086a (stale) Mirror 4: 0x6a8d086a (stale) Mirror 7: 0x6a8d086a (stale) Mirror 8: 0x6a8d086a (stale) Mirror 10: 0xa3f67ce3 (stale) Mirror 11: 0x53f044d0 (stale) Mirror 12: 0x39a31dd8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b19): Mirror 2: 0x6a8d086a (stale) Mirror 4: 0x6a8d086a (stale) Mirror 7: 0x6a8d086a (stale) Mirror 8: 0x6a8d086a (stale) Mirror 10: 0xa3f67ce3 (stale) Mirror 11: 0x53f044d0 (stale) Mirror 12: 0x39a31dd8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b19: skipped 000838 1789584396.570791 trunc from 0x018103 to 0x01e4b9 (0x63b7 bytes) 000839 1789584396.577150 trunc from 0x022af1 to 0x023996 (0xea6 bytes) 000840 1789584396.581489 punch from 0x01a29f to 0x026cdc (0xca3e bytes) 000841 1789584396.586538 write 0x00719c thru 0x0107a8 (0x960d bytes) 000842 1789584396.605000 punch from 0x007e2d to 0x017368 (0xf53c bytes) 000843 1789584396.620996 punch from 0x01adf9 to 0x029be2 (0xedea bytes) 000844 1789584396.630131 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37b19): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b19): Mirror 2: 0x6a8d086a (stale) Mirror 4: 0x6a8d086a (stale) Mirror 7: 0x6a8d086a (stale) Mirror 8: 0x6a8d086a (stale) Mirror 10: 0xa3f67ce3 (stale) Mirror 11: 0x53f044d0 (stale) Mirror 12: 0x39a31dd8 (stale) CRC-32 checksum value for chunk [0, 0x37b19): Mirror 1: 0x651396f8 Mirror 13: 0x651396f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b19: skipped 000845 1789584397.889524 write 0x00a5f6 thru 0x0191d2 (0xebdd bytes) 000846 1789584398.017701 trunc from 0x037b19 to 0x027569 (0x105af bytes) 000847 1789584398.077815 read 0x005892 thru 0x0097a9 (0x3f18 bytes) 000848 1789584398.095396 trunc from 0x01bbce to 0x02653c (0xa96f bytes) 000849 1789584398.102466 trunc from 0x003997 to 0x006545 (0x2baf bytes) 000850 1789584398.112414 mapread 0x01d1bb thru 0x021758 (0x459e bytes) 000851 1789584398.147959 mapwrite 0x03026b thru 0x0311db (0xf71 bytes) 000852 1789584398.207258 mapwrite 0x039332 thru 0x03ffff (0x6cce bytes) 000854 1789584398.258555 read 0x00e9bf thru 0x017658 (0x8c9a bytes) 000856 1789584398.260736 mapwrite 0x0074a9 thru 0x01381a (0xc372 bytes) 000857 1789584398.286577 mapread 0x00240a thru 0x00e6a2 (0xc299 bytes) 000858 1789584398.303016 MIRROR_RESYNC 9 mirrors 000859 1789584401.600182 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000860 1789584402.277371 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000862 1789584402.753317 punch from 0x00544e to 0x00e601 (0x91b4 bytes) 000863 1789584402.820383 write 0x031be3 thru 0x03a8c4 (0x8ce2 bytes) 000864 1789584402.840719 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb57e279f (stale) Mirror 4: 0xb57e279f (stale) Mirror 7: 0xb57e279f (stale) Mirror 10: 0xb57e279f (stale) Mirror 11: 0xb57e279f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb57e279f (stale) Mirror 4: 0xb57e279f (stale) Mirror 7: 0xb57e279f (stale) Mirror 10: 0xb57e279f (stale) Mirror 11: 0xb57e279f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000865 1789584403.394960 write 0x03d7e9 thru 0x03ffff (0x2817 bytes) 000866 1789584403.409650 mapwrite 0x01f75b thru 0x02d286 (0xdb2c bytes) 000867 1789584403.490769 punch from 0x005b05 to 0x00c9b9 (0x6eb5 bytes) 000870 1789584403.503792 fallocate 0x03f2b0 thru 0x03ffff (0xd50 bytes) 000871 1789584403.513050 mapread 0x01c521 thru 0x021f87 (0x5a67 bytes) 000872 1789584403.514769 trunc from 0x040000 to 0x02db4f (0x124b0 bytes) 000873 1789584403.530667 trunc from 0x02db50 to 0x0132ff (0x1a850 bytes) 000874 1789584403.552696 write 0x01bd2b thru 0x0280cc (0xc3a2 bytes) 000875 1789584403.560532 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x280cd): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x280cd): Mirror 2: 0x9ee7b1d6 (stale) Mirror 4: 0x9ee7b1d6 (stale) Mirror 7: 0x9ee7b1d6 (stale) Mirror 10: 0x9ee7b1d6 (stale) Mirror 11: 0x9ee7b1d6 (stale) CRC-32 checksum value for chunk [0, 0x280cd): Mirror 1: 0x77979c45 Mirror 12: 0x77979c45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x280cd: skipped 000876 1789584404.678230 trunc from 0x00397f to 0x00c555 (0x8bd7 bytes) 000877 1789584404.764791 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x280cd): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x280cd): Mirror 2: 0x9ee7b1d6 (stale) Mirror 4: 0x9ee7b1d6 (stale) Mirror 7: 0x9ee7b1d6 (stale) Mirror 10: 0x9ee7b1d6 (stale) Mirror 11: 0x9ee7b1d6 (stale) Mirror 12: 0x77979c45 (stale) CRC-32 checksum value for chunk [0, 0x280cd): Mirror 1: 0x77979c45 Mirror 13: 0x77979c45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x280cd: skipped 000878 1789584405.818849 write 0x03f1bc thru 0x03ffff (0xe44 bytes) 000879 1789584405.902696 punch from 0x034963 to 0x03b125 (0x67c3 bytes) 000880 1789584405.909351 read 0x015511 thru 0x01a76e (0x525e bytes) 000881 1789584405.920192 read 0x00ea19 thru 0x01c719 (0xdd01 bytes) 000882 1789584405.934425 mapwrite 0x01ca3d thru 0x01ef0a (0x24ce bytes) 000883 1789584405.955199 fallocate 0x021fd6 thru 0x029c38 (0x7c63 bytes) 000884 1789584405.961357 read 0x0161e7 thru 0x01adc6 (0x4be0 bytes) 000885 1789584405.963240 read 0x0379ac thru 0x03fff0 (0x8645 bytes) 000886 1789584405.964484 trunc from 0x040000 to 0x01fb62 (0x2049d bytes) 000887 1789584406.011410 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fb63): Mirror 2: 0x31eaea22 (stale) Mirror 4: 0x31eaea22 (stale) Mirror 7: 0x31eaea22 (stale) Mirror 10: 0x31eaea22 (stale) Mirror 11: 0x31eaea22 (stale) Mirror 12: 0x67d9279c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fb63): Mirror 2: 0x31eaea22 (stale) Mirror 4: 0x31eaea22 (stale) Mirror 7: 0x31eaea22 (stale) Mirror 10: 0x31eaea22 (stale) Mirror 11: 0x31eaea22 (stale) Mirror 12: 0x67d9279c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fb63: skipped 000888 1789584406.681219 write 0x02bed3 thru 0x02d8c9 (0x19f7 bytes) 000889 1789584406.688241 fallocate 0x01cfaa thru 0x022b5c (0x5bb3 bytes) 000890 1789584406.696752 punch from 0x02813a to 0x02d8c9 (0x5790 bytes) 000891 1789584406.709522 trunc from 0x004bb4 to 0x008dab (0x41f8 bytes) 000892 1789584406.716241 trunc from 0x008e96 to 0x01844c (0xf5b7 bytes) 000893 1789584406.720225 punch from 0x01e010 to 0x02d7d6 (0xf7c7 bytes) 000894 1789584406.724757 mapwrite 0x033e7f thru 0x03ffff (0xc181 bytes) 000895 1789584406.870751 punch from 0x008730 to 0x015823 (0xd0f4 bytes) 000896 1789584406.878757 trunc from 0x040000 to 0x001434 (0x3ebcb bytes) 000897 1789584406.918767 mapwrite 0x026e97 thru 0x031ca7 (0xae11 bytes) 000898 1789584406.988295 mapwrite 0x02fef8 thru 0x030580 (0x689 bytes) 000899 1789584407.002360 mapread 0x003c06 thru 0x0058a6 (0x1ca1 bytes) 000901 1789584407.024691 trunc from 0x000409 to 0x001862 (0x145a bytes) 000902 1789584407.033676 punch from 0x031aae to 0x031ca7 (0x1fa bytes) 000903 1789584407.041728 trunc from 0x031ca8 to 0x024c5a (0xd04d bytes) 000904 1789584407.062373 fallocate 0x02b290 thru 0x032c87 (0x79f8 bytes) 000905 1789584407.076169 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32c88): Mirror 2: 0x8810d1ea (stale) Mirror 4: 0x8810d1ea (stale) Mirror 7: 0x8810d1ea (stale) Mirror 10: 0x8810d1ea (stale) Mirror 12: 0xa1aa9a5f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c88): Mirror 2: 0x8810d1ea (stale) Mirror 4: 0x8810d1ea (stale) Mirror 7: 0x8810d1ea (stale) Mirror 10: 0x8810d1ea (stale) Mirror 12: 0xa1aa9a5f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c88: skipped 000906 1789584407.671205 read 0x026077 thru 0x02aced (0x4c77 bytes) 000908 1789584407.689451 mapwrite 0x0248f4 thru 0x02ada2 (0x64af bytes) 000910 1789584407.751406 fallocate 0x011fe5 thru 0x015108 (0x3124 bytes) 000911 1789584407.769249 trunc from 0x005ad1 to 0x00c94d (0x6e7d bytes) 000912 1789584407.775844 trunc from 0x032c88 to 0x001c80 (0x31007 bytes) 000913 1789584407.813058 write 0x005d05 thru 0x00834d (0x2649 bytes) 000914 1789584407.814363 trunc from 0x0082d2 to 0x00834d (0x7c bytes) 000915 1789584407.825336 trunc from 0x000bd4 to 0x0064e4 (0x5911 bytes) 000917 1789584407.840999 read 0x0074a2 thru 0x00834d (0xeac bytes) 000918 1789584407.877757 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x834e): Mirror 2: 0x7fb8da38 (stale) Mirror 4: 0x7fb8da38 (stale) Mirror 7: 0x7fb8da38 (stale) Mirror 10: 0x7fb8da38 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x834e): Mirror 2: 0x7fb8da38 (stale) Mirror 4: 0x7fb8da38 (stale) Mirror 7: 0x7fb8da38 (stale) Mirror 10: 0x7fb8da38 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x834e: skipped 000919 1789584408.258958 fallocate 0x0284a4 thru 0x033b8d (0xb6ea bytes) 000920 1789584408.265817 write 0x01fd27 thru 0x02dcbd (0xdf97 bytes) 000921 1789584408.288961 mapread 0x029c8d thru 0x033b8d (0x9f01 bytes) 000922 1789584408.309492 read 0x012a70 thru 0x01c6a2 (0x9c33 bytes) 000923 1789584408.318846 write 0x002b78 thru 0x00b0af (0x8538 bytes) 000924 1789584408.333982 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33b8e): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b8e): Mirror 2: 0x9c9a9e90 (stale) Mirror 4: 0x9c9a9e90 (stale) Mirror 7: 0x9c9a9e90 (stale) Mirror 10: 0x9c9a9e90 (stale) CRC-32 checksum value for chunk [0, 0x33b8e): Mirror 1: 0xc0dad15b Mirror 11: 0xc0dad15b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b8e: skipped 000925 1789584409.215465 mapread 0x011c6d thru 0x01472c (0x2ac0 bytes) 000926 1789584409.261911 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33b8e): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b8e): Mirror 2: 0x9c9a9e90 (stale) Mirror 4: 0x9c9a9e90 (stale) Mirror 7: 0x9c9a9e90 (stale) Mirror 10: 0x9c9a9e90 (stale) CRC-32 checksum value for chunk [0, 0x33b8e): Mirror 1: 0xc0dad15b Mirror 11: 0xc0dad15b Mirror 12: 0xc0dad15b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b8e: skipped 000927 1789584410.526573 mapwrite 0x009905 thru 0x013def (0xa4eb bytes) 000928 1789584410.743204 mapread 0x0142cb thru 0x015590 (0x12c6 bytes) 000929 1789584410.749172 punch from 0x01c84f to 0x0293a6 (0xcb58 bytes) 000930 1789584410.761398 write 0x01f183 thru 0x02aae3 (0xb961 bytes) 000931 1789584410.799368 trunc from 0x033b8e to 0x035429 (0x189c bytes) 000932 1789584410.856197 fallocate 0x02b48f thru 0x0382d1 (0xce43 bytes) 000933 1789584410.862509 read 0x014e9d thru 0x020904 (0xba68 bytes) 000934 1789584410.870336 mapwrite 0x02e555 thru 0x03a900 (0xc3ac bytes) 000935 1789584410.927655 mapwrite 0x00277f thru 0x003179 (0x9fb bytes) 000936 1789584410.953720 punch from 0x02aee9 to 0x02feb1 (0x4fc9 bytes) 000937 1789584410.964711 mapread 0x02a9d2 thru 0x02e1d7 (0x3806 bytes) 000938 1789584410.977181 mapread 0x00e316 thru 0x0185a2 (0xa28d bytes) 000939 1789584410.978457 trunc from 0x03a901 to 0x01f95e (0x1afa2 bytes) 000940 1789584410.994616 fallocate 0x00d64f thru 0x01b640 (0xdff2 bytes) 000941 1789584411.001054 trunc from 0x00bcd6 to 0x01803a (0xc365 bytes) 000942 1789584411.006907 mapread 0x00e4b9 thru 0x01e3c6 (0xff0e bytes) 000943 1789584411.021814 trunc from 0x01f95f to 0x03d9d6 (0x1e078 bytes) 000944 1789584411.041993 punch from 0x027468 to 0x02e1fd (0x6d96 bytes) 000945 1789584411.050703 mapwrite 0x031ed5 thru 0x036df1 (0x4f1d bytes) 000946 1789584411.088034 mapread 0x025338 thru 0x030873 (0xb53c bytes) 000947 1789584411.098559 trunc from 0x01bb53 to 0x02b14c (0xf5fa bytes) 000948 1789584411.104685 read 0x01caaf thru 0x027976 (0xaec8 bytes) 000949 1789584411.124229 write 0x03ccad thru 0x03ffff (0x3353 bytes) 000950 1789584411.125887 fallocate 0x00e9f0 thru 0x0121ed (0x37fe bytes) 000951 1789584411.134281 mapwrite 0x03bc34 thru 0x03ffff (0x43cc bytes) 000952 1789584411.159589 fallocate 0x00e4bf thru 0x01b500 (0xd042 bytes) 000953 1789584411.171662 MIRROR_RESYNC 7 mirrors 000954 1789584413.653288 punch from 0x01e1ad to 0x0249cc (0x6820 bytes) 000955 1789584413.720995 write 0x01f951 thru 0x021495 (0x1b45 bytes) 000957 1789584413.733422 mapwrite 0x016adf thru 0x026ad6 (0xfff8 bytes) 000958 1789584413.801712 mapwrite 0x0030fc thru 0x008a72 (0x5977 bytes) 000959 1789584413.835787 fallocate 0x015c84 thru 0x015ce2 (0x5f bytes) 000960 1789584413.846173 MIRROR_RESYNC 7 mirrors 000961 1789584416.929286 mapwrite 0x000b0c thru 0x00506a (0x455f bytes) 000962 1789584417.035826 write 0x010b30 thru 0x01de43 (0xd314 bytes) 000963 1789584417.048825 trunc from 0x040000 to 0x00bef5 (0x3410a bytes) 000964 1789584417.080015 write 0x029aea thru 0x038f96 (0xf4ad bytes) 000965 1789584417.088721 MIRROR_RESYNC 7 mirrors 000967 1789584418.627789 read 0x011743 thru 0x015c2b (0x44e9 bytes) 000969 1789584418.644406 MIRROR_RESYNC 7 mirrors 000970 1789584419.526378 mapwrite 0x03b308 thru 0x03f976 (0x466f bytes) 000971 1789584419.612952 read 0x02e11d thru 0x03715f (0x9043 bytes) 000972 1789584419.621763 write 0x0320a6 thru 0x0375fa (0x5555 bytes) 000976 1789584419.632761 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f977): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f977): Mirror 2: 0xe579155e (stale) Mirror 4: 0xe579155e (stale) Mirror 7: 0xe579155e (stale) Mirror 10: 0xe579155e (stale) Mirror 11: 0xe579155e (stale) Mirror 12: 0xe579155e (stale) CRC-32 checksum value for chunk [0, 0x3f977): Mirror 1: 0xa25674e0 Mirror 13: 0xa25674e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f977: skipped 000978 1789584420.489660 read 0x00af83 thru 0x015b41 (0xabbf bytes) 000979 1789584420.506988 trunc from 0x03f977 to 0x021cd0 (0x1dca6 bytes) 000980 1789584420.594791 mapread 0x00d777 thru 0x00f9e2 (0x226c bytes) 000981 1789584420.614527 mapread 0x00cd52 thru 0x00f1a9 (0x2458 bytes) 000982 1789584420.615744 punch from 0x0093e1 to 0x00be7e (0x2a9e bytes) 000983 1789584420.621304 punch from 0x00ee1c to 0x01bda8 (0xcf8d bytes) 000984 1789584420.626815 write 0x01b50a thru 0x026ea7 (0xb99e bytes) 000985 1789584420.633834 mapwrite 0x013996 thru 0x016c57 (0x32c2 bytes) 000986 1789584420.668063 trunc from 0x003ec2 to 0x00c87c (0x89bb bytes) 000987 1789584420.674698 mapwrite 0x02a191 thru 0x032481 (0x82f1 bytes) 000988 1789584420.742636 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32482): Mirror 2: 0x5d14319a (stale) Mirror 4: 0x5d14319a (stale) Mirror 7: 0x5d14319a (stale) Mirror 11: 0x5d14319a (stale) Mirror 12: 0x5d14319a (stale) Mirror 13: 0xd8eee305 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32482): Mirror 2: 0x5d14319a (stale) Mirror 4: 0x5d14319a (stale) Mirror 7: 0x5d14319a (stale) Mirror 11: 0x5d14319a (stale) Mirror 12: 0x5d14319a (stale) Mirror 13: 0xd8eee305 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32482: skipped 000989 1789584421.200947 mapwrite 0x022896 thru 0x02f32a (0xca95 bytes) 000990 1789584421.250364 mapwrite 0x0396ec thru 0x03ffff (0x6914 bytes) 000991 1789584421.300498 trunc from 0x0146d0 to 0x018751 (0x4082 bytes) 000992 1789584421.306278 read 0x01062c thru 0x010764 (0x139 bytes) 000993 1789584421.311993 trunc from 0x040000 to 0x03afac (0x5053 bytes) 000994 1789584421.328836 trunc from 0x01216c to 0x018681 (0x6516 bytes) 000995 1789584421.338888 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3afad): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3afad): Mirror 2: 0x6d1a14f0 (stale) Mirror 4: 0x6d1a14f0 (stale) Mirror 7: 0x6d1a14f0 (stale) Mirror 11: 0x6d1a14f0 (stale) Mirror 12: 0x6d1a14f0 (stale) Mirror 13: 0x4199faad (stale) CRC-32 checksum value for chunk [0, 0x3afad): Mirror 1: 0x79dbeacf Mirror 14: 0x79dbeacf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3afad: skipped 000996 1789584422.210761 MIRROR_RESYNC 8 mirrors 000998 1789584424.001211 punch from 0x0062b9 to 0x00ca9d (0x67e5 bytes) 001000 1789584424.093037 mapwrite 0x01445b thru 0x01edf7 (0xa99d bytes) All operations completed A-OK! PASS 70a (109s) == sanity-flr test 71: check mirror extend parent fid ==== 14:47:11 (1789584431) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x63:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 14:47:23 (1789584443) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229719 s, 9.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890067 s, 11.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c3:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c4:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 14:47:33 (1789584453) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1176864 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5711264 .. done Extending file size to 3984448 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7670528 .. Extending file size to 4624352 .. Extending file size to 8468576 .. Extending file size to 8333472 .. Extending file size to 8402112 .. Extending file size to 8703232 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8843168 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9034304 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 61337 61339 61340 61342 61343 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) 192.168.202.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (92s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:49:05 (1789584545) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 8167584 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4912864 .. failed Extending file size to 7414464 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8304416 .. Extending file size to 8408800 .. Extending file size to 7781728 .. Extending file size to 8951648 .. Extending file size to 8379520 .. Extending file size to 8440672 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8760064 .. Extending file size to 8764736 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8792608 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9111712 .. Waiting 63000 63002 63003 63006 63007 failed 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) 192.168.202.134@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:50:23 (1789584623) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18632 3588388 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24388 3582632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43020 7171020 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1992 1269312 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18632 3588388 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24388 3582632 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43020 7171020 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0858935 s, 24.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195327 s, 10.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd7d76c33 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:50:41 (1789584641) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1058 s, 19.8 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 14:50:48 (1789584648) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d9:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:50:56 (1789584656) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0759664 s, 27.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:51:06 (1789584666) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104871 s, 20.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2db:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:51:16 (1789584676) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178541 s, 11.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:51:51 (1789584711) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762514 s, 13.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x59:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x58:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x59:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x58:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x59:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:52:44 (1789584764) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0813843 s, 25.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:53:37 (1789584817) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg234-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg234-server: Pool lustre.test_204f created Adding targets to pool oleg234-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg234-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138301 s, 15.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x5a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg234-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg234-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg234-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:54:52 (1789584892) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:55:00 (1789584900) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:55:09 (1789584909) oleg234-server: Pool lustre.test_206 created Adding targets to pool oleg234-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg234-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg234-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg234-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg234-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:55:37 (1789584937) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919847 s, 11.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270864 s, 38.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb6b9f60e Mirror 3: 0xb6b9f60e PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:55:48 (1789584948) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:55:52 (1789584952) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:55:55 (1789584955) Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0954779 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x15e0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1340000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1340000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1340000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xc90000 Read this (4 bytes): BBBB 192.168.202.134@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg234-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:56:38 (1789584998) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0245487 s, 334 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:56:46 (1789585006) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613301 s, 17.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1e5 lmm_fid: [0x200000405:0x1e5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1e5 lmm_fid: [0x200000405:0x1e5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x308:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611621 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 14:56:48.000000000 -0400 Modify: 2026-09-16 14:56:48.000000000 -0400 Change: 2026-09-16 14:56:50.000000000 -0400 Birth: 2026-09-16 14:56:48.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:56:55 (1789585015) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032978 s, 31.8 MB/s oleg234-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg234-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg234-server: O/280000401/d9/309: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:57:19 (1789585039) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.237995 s, 172 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0517271 s, 79.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 16 14:57 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ea:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 16 14:57 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:57:28 (1789585048) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:57:37 (1789585057) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1997 sec =========== 14:57:48 (1789585068) === sanity-flr: start cleanup 14:57:49 (1789585069) === === sanity-flr: finish cleanup 14:57:59 (1789585079) ===