-----============= acceptance-small: sanity-flr ============----- Tue Jun 16 06:24:38 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg123-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 06:24:58 (1781605498) === oleg123-client.virtnet: executing check_config_client /mnt/lustre oleg123-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg123-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90f144d77800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90f144d77800.idle_timeout=debug disable quota as required oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 06:25:43 (1781605543) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7452] == sanity-flr test 0a: lfs mirror create with -N option == 06:25:46 (1781605546) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:26:07 (1781605567) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:26:11 (1781605571) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:26:15 (1781605575) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:26:44 (1781605604) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:26:48 (1781605608) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 06:26:53 (1781605613) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff90f144d77800.stats=clear osc.lustre-OST0001-osc-ffff90f144d77800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:27:02 (1781605622) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:27:16 (1781605636) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:27:26 (1781605646) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:27:30 (1781605650) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:27:39 (1781605659) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 06:27:51 (1781605671) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:27:59 (1781605679) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:28:03 (1781605683) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (12s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:28:15 (1781605695) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:28:28 (1781605708) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:28:36 (1781605716) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 1.15486 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069806 s, 15.0 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:28:47 (1781605727) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121575 s, 17.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:28:56 (1781605736) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:29:08 (1781605748) Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90f144d77800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90f144d77800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90f144d77800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90f144d77800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg123-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:30:22 (1781605822) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.173498 s, 24.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90f144d77800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90f144d77800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:31:06 (1781605866) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90f144d77800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90f144d77800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90f144d77800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90f144d77800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg123-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (85s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:32:31 (1781605951) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.75316 s, 14.0 MB/s -rw-r--r-- 1 root root 52428800 Jun 16 06:32 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.753s user 0m0.000s sys 0m0.164s Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90f144d77800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90f144d77800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.473s user 0m0.000s sys 0m0.192s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90f144d77800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90f144d77800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.379s user 0m0.001s sys 0m0.202s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg123-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg123-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90f144d77800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90f144d77800.ost_server_uuid in FULL state after 0 sec PASS 33b (99s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:34:11 (1781606051) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:34:14 (1781606054) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:34:18 (1781606058) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:34:22 (1781606062) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 06:34:31 (1781606071) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.605601 s, 20.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:34:45 (1781606085) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339074 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034689 s, 30.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:34:53 (1781606093) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:35:02 (1781606102) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00132795 s, 771 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000984454 s, 1.0 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115891 s, 884 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114629 s, 893 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 06:35:13 (1781606113) create test files with size 15747994 .. b2e65adc312e365171b6d9f1fb6cce87 - 8d2b978e5734a7fb11f4961575c05a42 - 639e6cc0c4ecb3a132ccb8bc0f1d0dee - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg123-client.virtnet /mnt/lustre (opts:) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 06:35:39 (1781606139) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (20s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:35:59 (1781606159) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:36:08 (1781606168) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844861 s, 12.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198248 s, 52.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 06:36:17 (1781606177) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113906 s, 18.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.35104 s, 11.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228465 s, 9.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 06:36:34 (1781606194) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 06:36:37 (1781606197) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:36:41 (1781606201) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208774 s, 50.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 06:36:49 (1781606209) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:36:52 (1781606212) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:37:02 (1781606222) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.515097 s, 20.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg123-client.virtnet /mnt/lustre (opts:) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:37:14 (1781606234) llite.lustre-ffff90f1580b8000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.343396 s, 30.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg123-client.virtnet /mnt/lustre (opts:) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff90f159b31800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:37:28 (1781606248) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:38:07 (1781606287) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:38:42 (1781606322) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:38:46 (1781606326) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.329862 s, 9.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.208342 s, 15.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5aaecdf4 (nosync,stale) Mirror 2: 0x5aaecdf4 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 0d3dc8e0e679feb131cf73bafa82eaa5 - mirror 1 checksum: 1315909c5c62efd9aad4f5e8d6a85c37 - mirror 2 checksum: 0d3dc8e0e679feb131cf73bafa82eaa5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-16T06:38:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-16T06:38:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0d3dc8e0e679feb131cf73bafa82eaa5 - mirror 1 checksum: 1315909c5c62efd9aad4f5e8d6a85c37 - mirror 2 checksum: 1315909c5c62efd9aad4f5e8d6a85c37 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-16T06:38:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-16T06:38:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:38:58 (1781606338) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477885 s, 21.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:39:07 (1781606347) Failing mds1 on oleg123-server Stopping /mnt/lustre-mds1 (opts:) on oleg123-server 06:39:20 (1781606360) shut down facet: mds1 facet_host: oleg123-server facet_failover_host: oleg123-server Failover mds1 to oleg123-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 06:39:43 (1781606383) targets are mounted 06:39:43 (1781606383) facet_failover done oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284066 s, 36.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x17 lmm_fid: [0x200000404:0x17:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:40:07 (1781606407) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0501854 s, 408 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0287902 s, 711 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:40:17 (1781606417) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0873235 s, 235 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0100313 s, 2.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:40:27 (1781606427) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0151926 s, 1.1 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:40:34 (1781606434) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0540646 s, 379 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:40:42 (1781606442) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00498946 s, 821 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0415591 s, 98.6 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:40:50 (1781606450) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x30:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:41:12 (1781606472) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:41:39 (1781606499) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x61:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:42:01 (1781606521) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:42:23 (1781606543) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (5s) == sanity-flr test 70: mirror create and split race ====== 06:42:28 (1781606548) mirror create pid 55328 mirror split pid 55329 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 55328 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 55329 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 06:43:35 (1781606615) Seed set to 6616 skipping zero length zero range truncating to largest ever: 0x96f3 000002 1781606617.157922 trunc from 00000000 to 0x0096f2 (0x96f3 bytes) 000003 1781606617.199242 mapread 0x006dff thru 0x0096f2 (0x28f4 bytes) 000004 1781606617.246064 punch from 0x00392a to 0x0096f2 (0x5dc9 bytes) 000006 1781606617.266545 trunc from 0x0096f3 to 0x00555d (0x4195 bytes) 000007 1781606617.298889 trunc from 0x002dfe to 0x00555d (0x2760 bytes) 000008 1781606617.315751 punch from 0x00327f to 0x00555d (0x22df bytes) 000009 1781606617.333923 write 0x019203 thru 0x01e3ce (0x51cc bytes) 000010 1781606617.340634 trunc from 0x00102b to 0x002601 (0x15d7 bytes) 000011 1781606617.345746 write 0x036d6b thru 0x03bf7a (0x5210 bytes) 000014 1781606617.350713 mapwrite 0x012d1a thru 0x019234 (0x651b bytes) 000015 1781606617.402997 punch from 0x032ac9 to 0x03bf7a (0x94b2 bytes) 000016 1781606617.415926 read 0x030482 thru 0x03806d (0x7bec bytes) truncating to largest ever: 0x2a556 000017 1781606617.430079 trunc from 0x03bf7b to 0x02a555 (0x11a25 bytes) 000018 1781606617.452278 trunc from 0x023f42 to 0x02a555 (0x6614 bytes) 000020 1781606617.459841 mapread 0x023cf5 thru 0x027321 (0x362d bytes) 000021 1781606617.470876 punch from 0x01b5ce to 0x02867e (0xd0b1 bytes) 000022 1781606617.478046 mapread 0x01f3f6 thru 0x02a555 (0xb160 bytes) 000023 1781606617.493327 trunc from 0x0125bd to 0x018cc1 (0x6705 bytes) fallocating to largest ever: 0x3f5dc 000024 1781606617.499786 fallocate 0x03cbe0 thru 0x03f5db (0x29fc bytes) 000025 1781606617.506127 mapread 0x00f68f thru 0x014544 (0x4eb6 bytes) 000026 1781606617.515322 trunc from 0x02a556 to 0x00a28a (0x202cb bytes) 000027 1781606617.534731 read 0x0054ab thru 0x008016 (0x2b6c bytes) 000028 1781606617.547613 read 0x005004 thru 0x00a28a (0x5287 bytes) 000029 1781606617.548512 mapwrite 0x0107fb thru 0x0130bb (0x28c1 bytes) 000030 1781606617.593081 write 0x00670f thru 0x011571 (0xae63 bytes) 000031 1781606617.595921 read 0x0114ae thru 0x0130bb (0x1c0e bytes) 000032 1781606617.596931 trunc from 0x000a41 to 0x00c7f2 (0xbdb2 bytes) 000033 1781606617.612968 trunc from 0x00ce5b to 0x0130bb (0x6261 bytes) 000034 1781606617.621243 punch from 0x004357 to 0x00fdfc (0xbaa6 bytes) 000035 1781606617.627698 mapwrite 0x03339b thru 0x03e5eb (0xb251 bytes) 000037 1781606617.683488 mapread 0x034350 thru 0x03b355 (0x7006 bytes) 000039 1781606617.686891 read 0x015a8e thru 0x024cea (0xf25d bytes) 000040 1781606617.688302 punch from 0x03d47e to 0x03e5eb (0x116e bytes) 000041 1781606617.693882 mapread 0x02a770 thru 0x034a6c (0xa2fd bytes) 000042 1781606617.696169 read 0x003172 thru 0x00e64b (0xb4da bytes) 000043 1781606617.698020 trunc from 0x00e023 to 0x01563d (0x761b bytes) 000044 1781606617.703007 punch from 0x031584 to 0x03e5eb (0xd068 bytes) 000045 1781606617.708690 mapwrite 0x007dcc thru 0x014c62 (0xce97 bytes) 000046 1781606617.745707 punch from 0x03a708 to 0x03e5eb (0x3ee4 bytes) 000047 1781606617.755930 MIRROR_EXTEND to 2 mirrors 000048 1781606618.366490 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) 000049 1781606618.509557 fallocate 0x032c4c thru 0x03cb43 (0x9ef8 bytes) 000050 1781606618.537543 trunc from 0x03e5ec to 0x01f864 (0x1ed87 bytes) 000051 1781606618.620744 write 0x02a6e5 thru 0x035845 (0xb161 bytes) 000052 1781606618.629096 write 0x022903 thru 0x030bfa (0xe2f8 bytes) 000053 1781606618.651653 mapwrite 0x009617 thru 0x00f69e (0x6088 bytes) 000054 1781606618.691958 punch from 0x008cd7 to 0x010bec (0x7f16 bytes) 000055 1781606618.698645 write 0x024263 thru 0x02d10c (0x8eaa bytes) 000056 1781606618.701513 read 0x02b9a8 thru 0x035845 (0x9e9e bytes) 000057 1781606618.703305 mapwrite 0x0050ee thru 0x010ba8 (0xbabb bytes) 000058 1781606618.735585 trunc from 0x035846 to 0x014a20 (0x20e25 bytes) 000060 1781606618.788037 trunc from 0x014a21 to 0x003a4d (0x10fd3 bytes) 000062 1781606618.805880 mapwrite 0x00c32f thru 0x00e48b (0x215d bytes) 000063 1781606618.867539 mapread 0x009482 thru 0x00e48b (0x500a bytes) 000065 1781606618.870369 trunc from 0x0050f4 to 0x0091ce (0x40db bytes) 000067 1781606618.875827 mapwrite 0x03300b thru 0x03ebe9 (0xbbdf bytes) 000068 1781606618.973530 punch from 0x01660c to 0x01d0de (0x6ad3 bytes) 000070 1781606618.982546 write 0x02e8c2 thru 0x036b13 (0x8252 bytes) 000072 1781606618.988492 read 0x00a5af thru 0x00f739 (0x518b bytes) 000073 1781606619.024482 read 0x030e4a thru 0x03445e (0x3615 bytes) 000075 1781606619.025749 mapread 0x002154 thru 0x010ea8 (0xed55 bytes) 000076 1781606619.031725 write 0x0345a9 thru 0x034d98 (0x7f0 bytes) truncating to largest ever: 0x3ffd6 000077 1781606619.032802 trunc from 0x03ebea to 0x03ffd5 (0x13ec bytes) 000078 1781606619.082994 punch from 0x032a62 to 0x036dd8 (0x4377 bytes) 000079 1781606619.120789 trunc from 0x020dad to 0x02313e (0x2392 bytes) 000080 1781606619.137329 mapwrite 0x02babb thru 0x037d8b (0xc2d1 bytes) 000081 1781606619.162965 mapwrite 0x018f82 thru 0x0257ec (0xc86b bytes) 000082 1781606619.182059 read 0x004a69 thru 0x00daf9 (0x9091 bytes) 000083 1781606619.183990 trunc from 0x03bb89 to 0x03ffd5 (0x444d bytes) 000084 1781606619.189532 read 0x01dbdf thru 0x021d8b (0x41ad bytes) 000085 1781606619.190469 punch from 0x00f621 to 0x010af8 (0x14d8 bytes) 000086 1781606619.199927 MIRROR_EXTEND to 2 mirrors 000087 1781606619.933500 punch from 0x03356c to 0x033676 (0x10b bytes) 000088 1781606619.978849 mapread 0x031bde thru 0x03ffd5 (0xe3f8 bytes) 000089 1781606620.006024 fallocate 0x025370 thru 0x031a9e (0xc72f bytes) 000091 1781606620.014323 punch from 0x009a3c to 0x017163 (0xd728 bytes) 000092 1781606620.030670 mapread 0x016e8b thru 0x023877 (0xc9ed bytes) 000093 1781606620.058982 mapread 0x00c42a thru 0x011ef6 (0x5acd bytes) 000094 1781606620.070944 trunc from 0x03ffd6 to 0x02d65d (0x12978 bytes) 000095 1781606620.133781 write 0x0110c2 thru 0x015e5f (0x4d9e bytes) 000096 1781606620.135091 mapread 0x012263 thru 0x021f1c (0xfcba bytes) 000097 1781606620.142923 mapwrite 0x020a61 thru 0x0261a6 (0x5746 bytes) 000098 1781606620.169413 mapwrite 0x00695a thru 0x00d072 (0x6719 bytes) 000099 1781606620.190108 punch from 0x002aed to 0x0061de (0x36f2 bytes) 000101 1781606620.196692 mapwrite 0x03f1fe thru 0x03ffff (0xe02 bytes) 000102 1781606620.255010 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb0c80406 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x27de6ef3 Mirror 3: 0x27de6ef3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000103 1781606621.094022 read 0x02dc24 thru 0x02f49a (0x1877 bytes) 000104 1781606621.105792 mapwrite 0x02d327 thru 0x03390a (0x65e4 bytes) 000105 1781606621.210238 MIRROR_RESYNC 3 mirrors 000106 1781606622.170939 mapread 0x012ee2 thru 0x015374 (0x2493 bytes) 000108 1781606622.190714 write 0x030f8d thru 0x033985 (0x29f9 bytes) 000109 1781606622.226017 write 0x02c484 thru 0x032484 (0x6001 bytes) 000110 1781606622.233283 mapread 0x03a2cd thru 0x03ffff (0x5d33 bytes) 000111 1781606622.239697 mapread 0x03ce99 thru 0x03ffff (0x3167 bytes) 000112 1781606622.240398 fallocate 0x0240b8 thru 0x026d2d (0x2c76 bytes) 000113 1781606622.246383 mapread 0x006ade thru 0x00b665 (0x4b88 bytes) 000114 1781606622.254721 mapwrite 0x0333e8 thru 0x03d4ed (0xa106 bytes) 000115 1781606622.272889 read 0x00e623 thru 0x0112ea (0x2cc8 bytes) 000116 1781606622.277692 trunc from 0x036908 to 0x03e09d (0x7796 bytes) 000118 1781606622.284517 fallocate 0x02d724 thru 0x03b81c (0xe0f9 bytes) 000119 1781606622.289593 mapwrite 0x00e051 thru 0x011a1a (0x39ca bytes) 000120 1781606622.306241 trunc from 0x040000 to 0x036e1a (0x91e5 bytes) 000121 1781606622.327831 read 0x017789 thru 0x01a33b (0x2bb3 bytes) 000122 1781606622.338896 mapwrite 0x01a111 thru 0x01e5ad (0x449d bytes) 000123 1781606622.367293 trunc from 0x02fa2b to 0x035c4b (0x6221 bytes) 000124 1781606622.384541 read 0x00e2b5 thru 0x01355e (0x52aa bytes) 000125 1781606622.385297 trunc from 0x00b2df to 0x00cdb4 (0x1ad6 bytes) 000126 1781606622.396831 mapwrite 0x013521 thru 0x017119 (0x3bf9 bytes) 000127 1781606622.429859 write 0x03e00c thru 0x03ffff (0x1ff4 bytes) 000128 1781606622.443899 write 0x0361c1 thru 0x03f1cc (0x900c bytes) 000129 1781606622.444852 punch from 0x005b84 to 0x00ad82 (0x51ff bytes) 000130 1781606622.453534 mapwrite 0x003aae thru 0x006cff (0x3252 bytes) 000131 1781606622.491803 trunc from 0x040000 to 0x013a0f (0x2c5f0 bytes) 000132 1781606622.511718 punch from 0x005950 to 0x005e1c (0x4cd bytes) 000133 1781606622.517568 mapwrite 0x00e71d thru 0x01c436 (0xdd1a bytes) 000134 1781606622.596439 write 0x03ace8 thru 0x03af6f (0x288 bytes) 000135 1781606622.597729 trunc from 0x03af70 to 0x03f50d (0x459e bytes) 000136 1781606622.620977 trunc from 0x03f50e to 0x00c345 (0x331c8 bytes) fallocating to largest ever: 0x40000 000137 1781606622.637977 fallocate 0x03f60b thru 0x03ffff (0x9f5 bytes) 000138 1781606622.653107 MIRROR_RESYNC 3 mirrors 000139 1781606623.425291 fallocate 0x037e68 thru 0x03ffff (0x8198 bytes) 000140 1781606623.460022 punch from 0x018323 to 0x0194eb (0x11c9 bytes) 000141 1781606623.469583 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa430a61f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa430a61f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000142 1781606623.750093 trunc from 0x004bca to 0x010fae (0xc3e5 bytes) 000143 1781606623.767728 punch from 0x034ed0 to 0x039dbe (0x4eef bytes) 000144 1781606623.773282 mapread 0x01b355 thru 0x01ccb4 (0x1960 bytes) 000145 1781606623.787954 mapread 0x00146e thru 0x00d2f6 (0xbe89 bytes) 000146 1781606623.801691 read 0x03285a thru 0x03407f (0x1826 bytes) 000147 1781606623.807849 read 0x012eb4 thru 0x018976 (0x5ac3 bytes) 000149 1781606623.824884 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa430a61f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf75b2386 Mirror 4: 0xf75b2386 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000150 1781606624.582419 trunc from 0x01dfa1 to 0x01f47d (0x14dd bytes) 000151 1781606624.625779 write 0x03c26a thru 0x03ffff (0x3d96 bytes) 000152 1781606624.636840 fallocate 0x035778 thru 0x03a3b2 (0x4c3b bytes) 000153 1781606624.642334 mapread 0x00cdf8 thru 0x014592 (0x779b bytes) 000154 1781606624.659915 write 0x00c299 thru 0x01352a (0x7292 bytes) 000155 1781606624.661481 mapread 0x011656 thru 0x01a14b (0x8af6 bytes) 000156 1781606624.672724 punch from 0x0351ef to 0x039e98 (0x4caa bytes) 000157 1781606624.677540 mapwrite 0x015276 thru 0x01785f (0x25ea bytes) 000158 1781606624.694456 trunc from 0x040000 to 0x00edc8 (0x31237 bytes) 000159 1781606624.718714 mapwrite 0x019521 thru 0x025c37 (0xc717 bytes) 000160 1781606624.779119 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25c38): Mirror 4: 0x99e619ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c38): Mirror 4: 0x99e619ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c38: skipped 000161 1781606625.026855 mapwrite 0x03c987 thru 0x03ffff (0x3679 bytes) 000162 1781606625.078972 write 0x03f457 thru 0x03ffff (0xba9 bytes) 000165 1781606625.082618 mapread 0x00b0f6 thru 0x011ef9 (0x6e04 bytes) 000167 1781606625.092081 punch from 0x0268ea to 0x034c1c (0xe333 bytes) 000168 1781606625.096720 punch from 0x00619b to 0x00a0cc (0x3f32 bytes) 000169 1781606625.101321 write 0x00c18a thru 0x00f06e (0x2ee5 bytes) 000170 1781606625.102440 read 0x00c1c6 thru 0x00f184 (0x2fbf bytes) 000171 1781606625.103050 punch from 0x03de65 to 0x03ffff (0x219b bytes) 000172 1781606625.117670 mapread 0x00c68a thru 0x00ec28 (0x259f bytes) 000174 1781606625.119763 trunc from 0x009ae2 to 0x018d26 (0xf245 bytes) 000176 1781606625.125642 fallocate 0x01805b thru 0x01d948 (0x58ee bytes) 000177 1781606625.130763 trunc from 0x040000 to 0x03cfdf (0x3020 bytes) 000178 1781606625.153943 fallocate 0x032e96 thru 0x03ae59 (0x7fc4 bytes) 000179 1781606625.158719 trunc from 0x03cfe0 to 0x01228b (0x2ad54 bytes) 000180 1781606625.192929 write 0x0099c1 thru 0x00c2c8 (0x2908 bytes) 000181 1781606625.202892 fallocate 0x00afb2 thru 0x01067e (0x56cd bytes) 000182 1781606625.212869 mapread 0x005d92 thru 0x00ffe8 (0xa257 bytes) 000184 1781606625.222555 mapread 0x001852 thru 0x00bbfe (0xa3ad bytes) 000185 1781606625.224703 mapread 0x009ff3 thru 0x010a1c (0x6a2a bytes) 000186 1781606625.226239 mapwrite 0x011b21 thru 0x01e4e3 (0xc9c3 bytes) 000187 1781606625.284981 mapwrite 0x03ba80 thru 0x03ffff (0x4580 bytes) 000188 1781606625.334985 trunc from 0x023ddd to 0x030990 (0xcbb4 bytes) 000189 1781606625.340518 fallocate 0x000b79 thru 0x010743 (0xfbcb bytes) 000191 1781606625.346397 mapread 0x017985 thru 0x022190 (0xa80c bytes) 000192 1781606625.358307 read 0x006a67 thru 0x011044 (0xa5de bytes) 000194 1781606625.359530 write 0x00593d thru 0x011d4e (0xc412 bytes) 000196 1781606625.362836 trunc from 0x009bb9 to 0x012e4d (0x9295 bytes) 000197 1781606625.385079 MIRROR_RESYNC 2 mirrors 000198 1781606626.058258 MIRROR_EXTEND to 3 mirrors 000199 1781606626.809476 write 0x03c5fe thru 0x03ffff (0x3a02 bytes) 000200 1781606626.852510 fallocate 0x00e323 thru 0x00ece2 (0x9c0 bytes) 000201 1781606626.857505 trunc from 0x02ffba to 0x03a3d3 (0xa41a bytes) 000202 1781606626.869562 mapread 0x00cb34 thru 0x0162a7 (0x9774 bytes) 000203 1781606626.897673 trunc from 0x040000 to 0x011fd1 (0x2e02e bytes) 000204 1781606626.941898 punch from 0x00d32f to 0x01167b (0x434d bytes) 000205 1781606626.950647 fallocate 0x005464 thru 0x005b2b (0x6c8 bytes) 000206 1781606626.970259 mapwrite 0x03d15e thru 0x03ffff (0x2ea2 bytes) 000207 1781606627.034244 trunc from 0x040000 to 0x024c23 (0x1b3dc bytes) 000208 1781606627.071381 trunc from 0x014e61 to 0x01a247 (0x53e7 bytes) 000209 1781606627.093901 trunc from 0x01ee5a to 0x0215a3 (0x274a bytes) 000210 1781606627.104689 fallocate 0x0060b3 thru 0x0073c6 (0x1314 bytes) 000211 1781606627.124739 mapwrite 0x03a376 thru 0x03ffff (0x5c8a bytes) 000212 1781606627.212835 write 0x03979b thru 0x03e67c (0x4ee2 bytes) 000213 1781606627.214103 write 0x0042aa thru 0x010f2c (0xcc83 bytes) 000214 1781606627.235841 fallocate 0x00a92b thru 0x016fe6 (0xc6bc bytes) 000215 1781606627.254371 read 0x03e297 thru 0x03ffff (0x1d69 bytes) 000216 1781606627.255009 write 0x01f968 thru 0x0219b7 (0x2050 bytes) 000217 1781606627.294352 MIRROR_RESYNC 3 mirrors 000218 1781606628.506735 read 0x0226a7 thru 0x030232 (0xdb8c bytes) 000219 1781606628.522521 trunc from 0x040000 to 0x0184fd (0x27b02 bytes) 000220 1781606628.600891 mapread 0x01396b thru 0x0184fd (0x4b93 bytes) 000221 1781606628.655256 read 0x00877a thru 0x011ad1 (0x9358 bytes) 000222 1781606628.684222 mapread 0x001c3a thru 0x00aca9 (0x9070 bytes) 000223 1781606628.687720 trunc from 0x0184fe to 0x03bb7a (0x2367d bytes) 000224 1781606628.753126 trunc from 0x00abc7 to 0x01724b (0xc685 bytes) 000225 1781606628.763701 read 0x02a4c3 thru 0x0357b0 (0xb2ee bytes) 000226 1781606628.786864 write 0x03f20c thru 0x03ffff (0xdf4 bytes) 000227 1781606628.787957 punch from 0x039380 to 0x03ffff (0x6c80 bytes) 000228 1781606628.812650 mapwrite 0x03db4b thru 0x03ffff (0x24b5 bytes) 000229 1781606628.833712 mapwrite 0x02886a thru 0x036e86 (0xe61d bytes) 000230 1781606628.869309 punch from 0x00a955 to 0x00edcf (0x447b bytes) 000231 1781606628.876356 mapread 0x015377 thru 0x01e98d (0x9617 bytes) 000232 1781606628.877984 mapwrite 0x0385a8 thru 0x03ffff (0x7a58 bytes) 000233 1781606628.917438 mapwrite 0x01828c thru 0x023fad (0xbd22 bytes) 000234 1781606628.939575 mapwrite 0x004d9c thru 0x00efa6 (0xa20b bytes) 000235 1781606628.985812 trunc from 0x040000 to 0x03025c (0xfda3 bytes) 000236 1781606629.009743 trunc from 0x017d05 to 0x01ab1f (0x2e1b bytes) 000237 1781606629.016988 trunc from 0x006008 to 0x009c9e (0x3c97 bytes) 000238 1781606629.022754 trunc from 0x03025d to 0x0372a5 (0x7049 bytes) 000240 1781606629.042415 fallocate 0x0351e3 thru 0x03a366 (0x5184 bytes) 000241 1781606629.048825 mapwrite 0x022f9e thru 0x024772 (0x17d5 bytes) 000242 1781606629.067098 mapwrite 0x039119 thru 0x03ffff (0x6ee7 bytes) 000244 1781606629.165834 mapwrite 0x03f421 thru 0x03ffff (0xbdf bytes) 000245 1781606629.179824 mapread 0x03a4de thru 0x03ffff (0x5b22 bytes) 000246 1781606629.181026 punch from 0x00d6e4 to 0x00e694 (0xfb1 bytes) 000248 1781606629.187783 fallocate 0x00814e thru 0x016d83 (0xec36 bytes) 000249 1781606629.193328 punch from 0x01bf39 to 0x025db7 (0x9e7f bytes) 000251 1781606629.200082 trunc from 0x040000 to 0x02350c (0x1caf3 bytes) 000252 1781606629.220475 mapwrite 0x01e350 thru 0x02c45a (0xe10b bytes) 000253 1781606629.326489 trunc from 0x01b3a8 to 0x021c6b (0x68c4 bytes) 000254 1781606629.332594 trunc from 0x007622 to 0x01205a (0xaa39 bytes) 000255 1781606629.337488 punch from 0x002d4b to 0x008859 (0x5b0f bytes) 000256 1781606629.343030 trunc from 0x02c45b to 0x021fa3 (0xa4b7 bytes) 000258 1781606629.361597 trunc from 0x021fa4 to 0x00d217 (0x14d8c bytes) 000259 1781606629.376981 trunc from 0x00ae50 to 0x00d217 (0x23c8 bytes) 000260 1781606629.382619 write 0x033aa8 thru 0x03fe98 (0xc3f1 bytes) 000262 1781606629.386301 punch from 0x00fc57 to 0x01a44d (0xa7f7 bytes) 000263 1781606629.390749 fallocate 0x001ad2 thru 0x005d33 (0x4262 bytes) 000264 1781606629.396386 mapwrite 0x00fc66 thru 0x01755e (0x78f9 bytes) 000265 1781606629.461318 fallocate 0x007346 thru 0x012332 (0xafed bytes) 000266 1781606629.466731 mapread 0x02d33b thru 0x0307bc (0x3482 bytes) 000267 1781606629.489300 mapwrite 0x03c5f2 thru 0x03d6f7 (0x1106 bytes) 000268 1781606629.502415 punch from 0x03a7fb to 0x03da09 (0x320f bytes) 000269 1781606629.508118 write 0x006417 thru 0x007035 (0xc1f bytes) 000270 1781606629.518313 trunc from 0x01207a to 0x01a4eb (0x8472 bytes) 000271 1781606629.525518 write 0x025c66 thru 0x02ce56 (0x71f1 bytes) 000272 1781606629.535686 punch from 0x025381 to 0x031580 (0xc200 bytes) 000273 1781606629.549076 trunc from 0x03a1f6 to 0x03fe98 (0x5ca3 bytes) 000274 1781606629.554290 punch from 0x0191b0 to 0x01bd7a (0x2bcb bytes) 000275 1781606629.559086 trunc from 0x0396d4 to 0x03fe98 (0x67c5 bytes) 000276 1781606629.564288 mapwrite 0x0375d7 thru 0x03ffff (0x8a29 bytes) 000277 1781606629.640511 fallocate 0x03bf56 thru 0x03d4d7 (0x1582 bytes) 000278 1781606629.645877 read 0x002684 thru 0x008d2f (0x66ac bytes) 000279 1781606629.680475 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb80816ab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb80816ab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000280 1781606629.930764 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xb80816ab (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3fff2ede Mirror 6: 0x3fff2ede lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000281 1781606630.689573 mapread 0x028c2c thru 0x03588f (0xcc64 bytes) 000282 1781606630.744808 mapread 0x03776a thru 0x03ffff (0x8896 bytes) 000284 1781606630.774533 trunc from 0x040000 to 0x016977 (0x29688 bytes) 000285 1781606630.831375 write 0x01b0b1 thru 0x01f9e2 (0x4932 bytes) 000286 1781606630.832755 read 0x00a0cc thru 0x00ea08 (0x493d bytes) 000287 1781606630.847108 trunc from 0x0004c6 to 0x006313 (0x5e4e bytes) 000288 1781606630.852023 fallocate 0x005b44 thru 0x01352d (0xd9ea bytes) 000289 1781606630.856583 mapread 0x007c4b thru 0x00b5b7 (0x396d bytes) 000290 1781606630.866531 mapwrite 0x00b14f thru 0x010c10 (0x5ac2 bytes) 000291 1781606630.887415 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1f9e3): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f9e3): Mirror 5: 0x18d3ea26 (stale) Mirror 6: 0x49026740 (stale) CRC-32 checksum value for chunk [0, 0x1f9e3): Mirror 1: 0x3400d427 Mirror 7: 0x3400d427 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f9e3: skipped 000292 1781606631.635936 trunc from 0x01f9e3 to 0x00345b (0x1c587 bytes) 000293 1781606631.686072 write 0x003886 thru 0x012898 (0xf013 bytes) 000294 1781606631.693676 mapread 0x005a97 thru 0x00c5c1 (0x6b2b bytes) 000295 1781606631.695606 mapread 0x00c731 thru 0x012898 (0x6168 bytes) 000296 1781606631.697345 trunc from 0x0127a0 to 0x012898 (0xf9 bytes) 000297 1781606631.717707 read 0x006a51 thru 0x0086bb (0x1c6b bytes) 000298 1781606631.723626 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12899): Mirror 5: 0x482adcab (stale) Mirror 7: 0x8d3d5d57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12899): Mirror 5: 0x482adcab (stale) Mirror 7: 0x8d3d5d57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12899: skipped 000299 1781606632.058475 write 0x02e05b thru 0x0334f5 (0x549b bytes) 000300 1781606632.066394 write 0x003456 thru 0x006c06 (0x37b1 bytes) 000301 1781606632.075363 punch from 0x0152ae to 0x01f2df (0xa032 bytes) 000302 1781606632.080463 punch from 0x00b076 to 0x016302 (0xb28d bytes) 000303 1781606632.094672 read 0x0329e4 thru 0x0334f5 (0xb12 bytes) 000304 1781606632.095524 mapwrite 0x003e30 thru 0x01171b (0xd8ec bytes) 000305 1781606632.175751 write 0x023aae thru 0x0261dc (0x272f bytes) 000306 1781606632.204694 punch from 0x01880b to 0x01adf4 (0x25ea bytes) 000307 1781606632.213668 read 0x006f6e thru 0x00b5cb (0x465e bytes) 000308 1781606632.215035 mapwrite 0x004c33 thru 0x0113c9 (0xc797 bytes) 000309 1781606632.240847 punch from 0x02d66a to 0x0334f5 (0x5e8c bytes) 000310 1781606632.247514 write 0x01f5fc thru 0x020f44 (0x1949 bytes) 000313 1781606632.266809 trunc from 0x0334f6 to 0x00ae11 (0x286e4 bytes) 000314 1781606632.284540 mapread 0x00857f thru 0x00ae11 (0x2893 bytes) 000315 1781606632.286305 read 0x002940 thru 0x00764b (0x4d0c bytes) 000316 1781606632.287528 fallocate 0x038271 thru 0x03ffff (0x7d8f bytes) 000317 1781606632.297311 MIRROR_RESYNC 3 mirrors 000318 1781606633.026378 write 0x00531c thru 0x014b52 (0xf837 bytes) 000319 1781606633.080383 trunc from 0x040000 to 0x0155b0 (0x2aa4f bytes) 000320 1781606633.116901 mapwrite 0x0135c7 thru 0x0171b4 (0x3bee bytes) 000321 1781606633.174974 trunc from 0x0171b5 to 0x038bd2 (0x21a1e bytes) 000322 1781606633.204743 MIRROR_RESYNC 3 mirrors 000323 1781606633.930798 mapwrite 0x003c1f thru 0x00adb4 (0x7196 bytes) 000324 1781606634.025783 mapread 0x030a01 thru 0x038bd2 (0x81d2 bytes) 000325 1781606634.035515 trunc from 0x038bd3 to 0x02c442 (0xc790 bytes) 000326 1781606634.061523 trunc from 0x027852 to 0x02a9ea (0x3199 bytes) 000327 1781606634.068319 mapwrite 0x003689 thru 0x0045f9 (0xf71 bytes) 000328 1781606634.080675 mapread 0x010377 thru 0x019280 (0x8f0a bytes) 000329 1781606634.090985 trunc from 0x019449 to 0x01c086 (0x2c3e bytes) 000330 1781606634.094685 fallocate 0x0217c7 thru 0x02dc56 (0xc490 bytes) 000331 1781606634.100798 punch from 0x013ba9 to 0x014f4e (0x13a6 bytes) 000332 1781606634.105838 trunc from 0x01c7e2 to 0x023760 (0x6f7f bytes) 000333 1781606634.110512 read 0x00e8d1 thru 0x01140f (0x2b3f bytes) 000334 1781606634.111419 read 0x00cdd6 thru 0x01bae9 (0xed14 bytes) 000335 1781606634.122537 punch from 0x02542e to 0x02969d (0x4270 bytes) 000336 1781606634.127644 mapwrite 0x02c737 thru 0x03c207 (0xfad1 bytes) 000337 1781606634.185073 mapread 0x024af3 thru 0x02829f (0x37ad bytes) 000338 1781606634.204493 fallocate 0x020b34 thru 0x026ee8 (0x63b5 bytes) 000339 1781606634.208031 fallocate 0x01dc5d thru 0x01e783 (0xb27 bytes) 000340 1781606634.215817 mapread 0x01912f thru 0x02457e (0xb450 bytes) 000341 1781606634.217005 mapwrite 0x035665 thru 0x03ffff (0xa99b bytes) 000342 1781606634.243962 trunc from 0x030690 to 0x03d8eb (0xd25c bytes) 000343 1781606634.248937 write 0x038c3e thru 0x03ca82 (0x3e45 bytes) 000344 1781606634.255627 trunc from 0x040000 to 0x02e179 (0x11e86 bytes) 000347 1781606634.266332 mapread 0x022dee thru 0x028902 (0x5b15 bytes) 000348 1781606634.267356 trunc from 0x02e17a to 0x01c0ea (0x1208f bytes) 000349 1781606634.282546 trunc from 0x013e73 to 0x01c0ea (0x8278 bytes) 000351 1781606634.288079 read 0x000062 thru 0x003037 (0x2fd6 bytes) 000352 1781606634.288585 write 0x001e3d thru 0x00bdd0 (0x9f94 bytes) 000353 1781606634.289713 trunc from 0x01c0eb to 0x009f9e (0x1214c bytes) 000354 1781606634.303235 mapwrite 0x012a91 thru 0x013e9c (0x140c bytes) 000355 1781606634.339553 mapwrite 0x01fec7 thru 0x023b4d (0x3c87 bytes) 000357 1781606634.383405 write 0x018277 thru 0x019d6a (0x1af4 bytes) 000358 1781606634.384366 fallocate 0x024a32 thru 0x02a4a5 (0x5a74 bytes) 000359 1781606634.388470 read 0x0147ec thru 0x017274 (0x2a89 bytes) 000360 1781606634.389207 punch from 0x010ba3 to 0x01f95f (0xedbd bytes) 000362 1781606634.397638 fallocate 0x014148 thru 0x01d695 (0x954e bytes) 000363 1781606634.404752 read 0x000a63 thru 0x00ad84 (0xa322 bytes) 000364 1781606634.406348 fallocate 0x01565e thru 0x01bc92 (0x6635 bytes) 000365 1781606634.412287 fallocate 0x025bcf thru 0x02e7bb (0x8bed bytes) 000366 1781606634.417512 mapread 0x019399 thru 0x01cb49 (0x37b1 bytes) 000367 1781606634.433648 punch from 0x0177da to 0x0182a1 (0xac8 bytes) 000368 1781606634.437666 trunc from 0x02e7bc to 0x035d9f (0x75e4 bytes) 000369 1781606634.452289 trunc from 0x035da0 to 0x01801e (0x1dd81 bytes) 000370 1781606634.465328 punch from 0x00c270 to 0x01801e (0xbdaf bytes) 000372 1781606634.469973 trunc from 0x01801f to 0x01a697 (0x2679 bytes) 000373 1781606634.483372 MIRROR_RESYNC 3 mirrors 000374 1781606635.079824 fallocate 0x02ae16 thru 0x037185 (0xc370 bytes) 000375 1781606635.131511 mapwrite 0x012c52 thru 0x01391a (0xcc9 bytes) 000376 1781606635.167312 read 0x00a9ea thru 0x01a54f (0xfb66 bytes) 000377 1781606635.185030 fallocate 0x035425 thru 0x03f8dc (0xa4b8 bytes) 000378 1781606635.193466 mapread 0x012fb6 thru 0x02068a (0xd6d5 bytes) 000379 1781606635.208478 punch from 0x00b942 to 0x00cd75 (0x1434 bytes) 000380 1781606635.215000 write 0x03a18a thru 0x03ff45 (0x5dbc bytes) 000381 1781606635.228114 trunc from 0x03ff46 to 0x03356a (0xc9db bytes) 000382 1781606635.289582 read 0x03047f thru 0x03356a (0x30ec bytes) 000383 1781606635.303842 mapwrite 0x008fd5 thru 0x0139da (0xaa06 bytes) 000384 1781606635.332747 mapread 0x0328d0 thru 0x03356a (0xc9b bytes) 000385 1781606635.339434 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3356b): Mirror 5: 0x8f49cdc7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3356b): Mirror 5: 0x8f49cdc7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3356b: skipped 000387 1781606635.611362 trunc from 0x03356b to 0x03f673 (0xc109 bytes) 000388 1781606635.648873 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f674): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f674): Mirror 5: 0x52e69b4 (stale) CRC-32 checksum value for chunk [0, 0x3f674): Mirror 1: 0x214ea6a8 Mirror 6: 0x214ea6a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f674: skipped 000389 1781606636.546754 trunc from 0x03f674 to 0x022a46 (0x1cc2d bytes) 000390 1781606636.610732 write 0x038be2 thru 0x039e4e (0x126d bytes) 000391 1781606636.613044 write 0x02f208 thru 0x0358c0 (0x66b9 bytes) 000392 1781606636.627214 write 0x02c977 thru 0x03380e (0x6e98 bytes) 000394 1781606636.635014 write 0x015ec0 thru 0x01a17c (0x42bd bytes) 000395 1781606636.651256 trunc from 0x039e4f to 0x02a8e4 (0xf56a bytes) 000396 1781606636.679846 fallocate 0x009d9f thru 0x00d1cb (0x342d bytes) 000397 1781606636.691867 read 0x002971 thru 0x00b81b (0x8eab bytes) 000398 1781606636.701624 read 0x01017c thru 0x019e33 (0x9cb8 bytes) 000399 1781606636.712729 trunc from 0x02a8e5 to 0x039ed5 (0xf5f1 bytes) 000400 1781606636.752458 trunc from 0x009e7a to 0x00e71d (0x48a4 bytes) 000402 1781606636.759127 punch from 0x039d03 to 0x039ed5 (0x1d3 bytes) 000403 1781606636.764666 mapwrite 0x03cc37 thru 0x03db72 (0xf3c bytes) 000404 1781606636.833013 punch from 0x0066bc to 0x014534 (0xde79 bytes) 000405 1781606636.843338 write 0x00da89 thru 0x01989f (0xbe17 bytes) 000406 1781606636.848845 read 0x027889 thru 0x02fa60 (0x81d8 bytes) 000408 1781606636.850304 fallocate 0x03193d thru 0x036005 (0x46c9 bytes) 000409 1781606636.854699 fallocate 0x039cbe thru 0x03a2ba (0x5fd bytes) 000410 1781606636.858415 fallocate 0x0350c5 thru 0x03664f (0x158b bytes) 000411 1781606636.866870 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3db73): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3db73): Mirror 5: 0x776d110b (stale) Mirror 6: 0x4aede83c (stale) CRC-32 checksum value for chunk [0, 0x3db73): Mirror 1: 0xc0c64d64 Mirror 7: 0xc0c64d64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3db73: skipped 000412 1781606638.052596 write 0x03ebae thru 0x03ffff (0x1452 bytes) 000413 1781606638.100386 trunc from 0x040000 to 0x01d5d7 (0x22a28 bytes) 000414 1781606638.123273 fallocate 0x02ea6e thru 0x0335e4 (0x4b77 bytes) 000415 1781606638.128983 fallocate 0x00d1e7 thru 0x01281b (0x5635 bytes) 000416 1781606638.133831 fallocate 0x02084d thru 0x02810b (0x78bf bytes) 000417 1781606638.138579 trunc from 0x002151 to 0x00abad (0x8a5d bytes) 000419 1781606638.144507 mapread 0x01693e thru 0x01c87f (0x5f42 bytes) 000420 1781606638.163361 punch from 0x00072c to 0x01048e (0xfd63 bytes) 000421 1781606638.169761 trunc from 0x0258b9 to 0x028fff (0x3747 bytes) 000422 1781606638.177494 trunc from 0x0335e5 to 0x012fce (0x20616 bytes) 000423 1781606638.206637 mapwrite 0x01d106 thru 0x01e7ad (0x16a8 bytes) 000424 1781606638.262360 trunc from 0x01e7ae to 0x03aef3 (0x1c746 bytes) 000425 1781606638.341526 punch from 0x01dad3 to 0x028595 (0xaac3 bytes) 000426 1781606638.356163 mapwrite 0x00ebae thru 0x0158b5 (0x6d08 bytes) 000428 1781606638.408984 trunc from 0x03aef4 to 0x03f316 (0x4423 bytes) 000429 1781606638.441709 trunc from 0x03f317 to 0x00746a (0x37eac bytes) 000431 1781606638.484651 trunc from 0x00746b to 0x031a0c (0x2a5a2 bytes) 000432 1781606638.540891 read 0x024095 thru 0x031a0c (0xd978 bytes) 000433 1781606638.550794 trunc from 0x01e03f to 0x02592a (0x78ec bytes) 000434 1781606638.558657 trunc from 0x031a0d to 0x037143 (0x5737 bytes) 000435 1781606638.576804 trunc from 0x0173af to 0x01be43 (0x4a95 bytes) 000436 1781606638.583087 fallocate 0x03f0ae thru 0x03ffff (0xf52 bytes) 000437 1781606638.590081 write 0x01bfee thru 0x0267d5 (0xa7e8 bytes) 000438 1781606638.597623 trunc from 0x0065b6 to 0x008b42 (0x258d bytes) 000439 1781606638.604562 fallocate 0x017ec1 thru 0x027b83 (0xfcc3 bytes) 000440 1781606638.610364 read 0x02f0dd thru 0x037143 (0x8067 bytes) 000441 1781606638.635957 mapwrite 0x01a779 thru 0x0242f3 (0x9b7b bytes) 000443 1781606638.661010 trunc from 0x02adf8 to 0x034243 (0x944c bytes) 000444 1781606638.666287 trunc from 0x01b14f to 0x0229fa (0x78ac bytes) 000447 1781606638.673432 read 0x01ad37 thru 0x028154 (0xd41e bytes) 000448 1781606638.684050 read 0x0279a5 thru 0x036578 (0xebd4 bytes) 000449 1781606638.684909 punch from 0x005b92 to 0x00e177 (0x85e6 bytes) 000450 1781606638.691360 write 0x021303 thru 0x025af0 (0x47ee bytes) 000451 1781606638.692734 mapwrite 0x01723b thru 0x01e35f (0x7125 bytes) 000452 1781606638.710065 fallocate 0x033a09 thru 0x03676a (0x2d62 bytes) 000453 1781606638.715956 mapread 0x01f676 thru 0x0219ab (0x2336 bytes) 000454 1781606638.717486 mapread 0x00e6bd thru 0x01e53c (0xfe80 bytes) 000455 1781606638.727607 punch from 0x01b3a6 to 0x01ccc8 (0x1923 bytes) 000456 1781606638.732626 mapwrite 0x030eb4 thru 0x03a6c6 (0x9813 bytes) 000457 1781606638.781214 mapwrite 0x0373d5 thru 0x039948 (0x2574 bytes) 000459 1781606638.799784 fallocate 0x03a330 thru 0x03ffff (0x5cd0 bytes) 000460 1781606638.805548 read 0x02d4bf thru 0x03bba3 (0xe6e5 bytes) 000461 1781606638.814691 mapwrite 0x0271c8 thru 0x02ec41 (0x7a7a bytes) 000462 1781606638.834524 fallocate 0x032d6a thru 0x03bf83 (0x921a bytes) 000463 1781606638.840592 trunc from 0x009824 to 0x00a352 (0xb2f bytes) 000464 1781606638.847122 fallocate 0x038502 thru 0x03ffff (0x7afe bytes) 000465 1781606638.852455 fallocate 0x037f73 thru 0x03d159 (0x51e7 bytes) 000466 1781606638.858503 punch from 0x036a9b to 0x037208 (0x76e bytes) 000467 1781606638.865505 mapread 0x00f15b thru 0x01b21c (0xc0c2 bytes) 000468 1781606638.866928 mapwrite 0x002d89 thru 0x0108b7 (0xdb2f bytes) 000469 1781606638.908320 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4736eebd (stale) Mirror 7: 0x9558a0c3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4736eebd (stale) Mirror 7: 0x9558a0c3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000470 1781606639.196640 trunc from 0x040000 to 0x01c545 (0x23aba bytes) 000471 1781606639.222569 write 0x023fb7 thru 0x02a933 (0x697d bytes) 000473 1781606639.226241 punch from 0x01f717 to 0x02a933 (0xb21d bytes) 000474 1781606639.243033 mapwrite 0x019c21 thru 0x01d7a9 (0x3b89 bytes) 000475 1781606639.283511 trunc from 0x02a934 to 0x03ee71 (0x1453e bytes) 000476 1781606639.301405 read 0x012745 thru 0x021565 (0xee21 bytes) 000477 1781606639.314439 MIRROR_RESYNC 3 mirrors 000478 1781606640.232228 MIRROR_RESYNC 3 mirrors 000479 1781606640.796108 fallocate 0x006926 thru 0x007363 (0xa3e bytes) 000480 1781606640.840709 read 0x02bf21 thru 0x02f392 (0x3472 bytes) 000481 1781606640.851073 write 0x003a8d thru 0x007d93 (0x4307 bytes) 000482 1781606640.862008 trunc from 0x0394a5 to 0x03ee71 (0x59cd bytes) 000483 1781606640.867869 punch from 0x035f97 to 0x0389d6 (0x2a40 bytes) 000484 1781606640.872340 fallocate 0x00d1a8 thru 0x01cde7 (0xfc40 bytes) 000485 1781606640.881101 trunc from 0x03ee72 to 0x0193ff (0x25a72 bytes) 000486 1781606640.939241 mapread 0x015b8d thru 0x016b68 (0xfdc bytes) 000487 1781606640.949234 write 0x023ac3 thru 0x031c4a (0xe188 bytes) 000488 1781606640.952022 trunc from 0x011944 to 0x01c578 (0xac35 bytes) 000489 1781606640.957597 read 0x01c8c2 thru 0x0238e7 (0x7026 bytes) 000490 1781606640.975232 trunc from 0x00c0d0 to 0x011742 (0x5673 bytes) 000491 1781606640.980876 write 0x03ee76 thru 0x03ffff (0x118a bytes) 000492 1781606640.982563 trunc from 0x037d7c to 0x03ffff (0x8284 bytes) 000493 1781606640.997676 read 0x035f53 thru 0x03ffff (0xa0ad bytes) 000495 1781606641.008778 mapwrite 0x0056fd thru 0x00d299 (0x7b9d bytes) 000496 1781606641.029431 trunc from 0x040000 to 0x00a88c (0x35773 bytes) 000497 1781606641.044377 trunc from 0x00a88d to 0x01d17e (0x128f2 bytes) 000499 1781606641.066030 mapread 0x017e51 thru 0x01d17e (0x532e bytes) 000500 1781606641.097532 trunc from 0x01d17f to 0x0156ee (0x7a90 bytes) 000501 1781606641.112087 mapread 0x01082e thru 0x0156ee (0x4ec1 bytes) 000502 1781606641.140197 mapread 0x0107b8 thru 0x011037 (0x880 bytes) 000503 1781606641.150668 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x156ef): Mirror 7: 0x5875c5fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x156ef): Mirror 7: 0x5875c5fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x156ef: skipped 000504 1781606641.428333 read 0x002b46 thru 0x00e08e (0xb549 bytes) 000505 1781606641.438958 read 0x013b03 thru 0x0156ee (0x1bec bytes) 000506 1781606641.447926 trunc from 0x0156ef to 0x000a33 (0x14cbb bytes) 000507 1781606641.470526 punch from 0x000722 to 0x000a33 (0x312 bytes) 000509 1781606641.479092 write 0x020964 thru 0x02d292 (0xc92f bytes) 000510 1781606641.480630 punch from 0x020e14 to 0x02d292 (0xc47f bytes) 000511 1781606641.502712 punch from 0x003dc2 to 0x0063f0 (0x262f bytes) 000512 1781606641.508600 mapwrite 0x0196f6 thru 0x023395 (0x9ca0 bytes) 000513 1781606641.546664 trunc from 0x02d293 to 0x007fa3 (0x252ef bytes) 000514 1781606641.578061 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) 000515 1781606641.704236 trunc from 0x006ea8 to 0x007fa3 (0x10fc bytes) 000516 1781606641.710111 mapwrite 0x00a607 thru 0x00c4e4 (0x1ede bytes) 000517 1781606641.767493 read 0x0021b4 thru 0x00c4e4 (0xa331 bytes) 000518 1781606641.774888 trunc from 0x00c4e5 to 0x03e80a (0x32326 bytes) 000519 1781606641.793549 mapread 0x03b689 thru 0x03e80a (0x3182 bytes) 000520 1781606641.817077 trunc from 0x021fb7 to 0x031058 (0xf0a2 bytes) 000521 1781606641.823874 fallocate 0x01e2d2 thru 0x02553b (0x726a bytes) 000522 1781606641.830484 trunc from 0x00c222 to 0x019a57 (0xd836 bytes) 000523 1781606641.838074 punch from 0x01cc4a to 0x01fcd5 (0x308c bytes) 000524 1781606641.843211 read 0x021430 thru 0x02bd20 (0xa8f1 bytes) 000525 1781606641.853411 mapread 0x004e18 thru 0x01341d (0xe606 bytes) 000526 1781606641.854809 trunc from 0x0200f8 to 0x02d770 (0xd679 bytes) 000527 1781606641.862128 read 0x011599 thru 0x011d91 (0x7f9 bytes) 000528 1781606641.862779 mapwrite 0x036753 thru 0x03ee52 (0x8700 bytes) 000530 1781606641.914000 write 0x012b23 thru 0x0195b1 (0x6a8f bytes) 000532 1781606641.915847 punch from 0x008649 to 0x00f064 (0x6a1c bytes) 000533 1781606641.929533 read 0x021be4 thru 0x022ce5 (0x1102 bytes) 000534 1781606641.941298 mapread 0x009747 thru 0x0153b2 (0xbc6c bytes) 000535 1781606641.942747 write 0x03c516 thru 0x03ffff (0x3aea bytes) 000536 1781606641.944734 write 0x0175d4 thru 0x01d446 (0x5e73 bytes) 000537 1781606641.947416 write 0x00c370 thru 0x015bf7 (0x9888 bytes) 000538 1781606641.949635 fallocate 0x03c347 thru 0x03ffff (0x3cb9 bytes) 000539 1781606641.956102 mapread 0x01ea25 thru 0x02c642 (0xdc1e bytes) 000541 1781606641.963449 punch from 0x039622 to 0x03ffff (0x69de bytes) 000542 1781606642.012950 read 0x00b9d5 thru 0x01664e (0xac7a bytes) 000544 1781606642.027515 MIRROR_EXTEND to 2 mirrors 000546 1781606642.876480 MIRROR_EXTEND to 3 mirrors 000547 1781606643.614413 mapread 0x024cc3 thru 0x033e67 (0xf1a5 bytes) 000549 1781606643.665946 fallocate 0x031a51 thru 0x0378d8 (0x5e88 bytes) 000550 1781606643.702376 fallocate 0x0213b4 thru 0x02b3de (0xa02b bytes) 000551 1781606643.711700 trunc from 0x040000 to 0x00f8a0 (0x3075f bytes) 000552 1781606643.731963 mapread 0x007515 thru 0x00f8a0 (0x838c bytes) 000553 1781606643.745178 trunc from 0x00f8a1 to 0x02133a (0x11a9a bytes) 000554 1781606643.768540 trunc from 0x020afd to 0x02133a (0x83e bytes) 000556 1781606643.774466 trunc from 0x006e9f to 0x00753b (0x69d bytes) 000557 1781606643.778913 fallocate 0x032665 thru 0x03d87b (0xb217 bytes) 000558 1781606643.784451 punch from 0x025440 to 0x026437 (0xff8 bytes) 000559 1781606643.789005 punch from 0x015078 to 0x02277e (0xd707 bytes) 000560 1781606643.794201 fallocate 0x00836e thru 0x016cb1 (0xe944 bytes) 000561 1781606643.799404 mapwrite 0x00790a thru 0x007fcb (0x6c2 bytes) 000562 1781606643.819066 mapwrite 0x02e3a6 thru 0x03d738 (0xf393 bytes) 000565 1781606643.873822 write 0x025cc5 thru 0x02cab5 (0x6df1 bytes) 000566 1781606643.880223 mapread 0x022352 thru 0x02682c (0x44db bytes) 000567 1781606643.890325 punch from 0x00d6b0 to 0x011e40 (0x4791 bytes) 000568 1781606643.895399 read 0x00cfd7 thru 0x012339 (0x5363 bytes) 000569 1781606643.906803 mapwrite 0x008ef0 thru 0x012f34 (0xa045 bytes) 000570 1781606643.926379 mapread 0x000e5d thru 0x003a43 (0x2be7 bytes) 000571 1781606643.933390 read 0x033f7c thru 0x03af24 (0x6fa9 bytes) 000572 1781606643.938882 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d87c): Mirror 3: 0x85823030 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d87c): Mirror 3: 0x85823030 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d87c: skipped 000573 1781606644.261834 trunc from 0x02b082 to 0x02b922 (0x8a1 bytes) 000574 1781606644.272263 mapread 0x02f2c2 thru 0x038246 (0x8f85 bytes) 000575 1781606644.292383 trunc from 0x00f455 to 0x010748 (0x12f4 bytes) 000576 1781606644.297586 trunc from 0x0318d4 to 0x03d87b (0xbfa8 bytes) 000577 1781606644.303984 trunc from 0x03d87c to 0x0061c0 (0x376bb bytes) 000578 1781606644.327617 fallocate 0x0373b4 thru 0x03a25d (0x2eaa bytes) 000579 1781606644.342701 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x61c1): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x61c1): Mirror 3: 0x3df983c6 (stale) CRC-32 checksum value for chunk [0, 0x61c1): Mirror 1: 0x3df983c6 Mirror 4: 0x3df983c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x61c1: skipped 000581 1781606645.237683 mapread 0x00007f thru 0x0015ad (0x152f bytes) 000582 1781606645.282107 trunc from 0x0061c1 to 0x0222d1 (0x1c111 bytes) 000583 1781606645.362545 trunc from 0x00de1f to 0x01c7fe (0xe9e0 bytes) 000584 1781606645.376385 punch from 0x019770 to 0x0205f2 (0x6e83 bytes) 000585 1781606645.390436 write 0x02a828 thru 0x0349fd (0xa1d6 bytes) 000586 1781606645.392571 trunc from 0x033a60 to 0x0340eb (0x68c bytes) 000587 1781606645.438332 MIRROR_RESYNC 3 mirrors 000588 1781606646.014450 write 0x00cf7b thru 0x01bac3 (0xeb49 bytes) 000589 1781606646.060044 write 0x031eca thru 0x037bf2 (0x5d29 bytes) 000590 1781606646.068290 mapwrite 0x03ee2d thru 0x03ffff (0x11d3 bytes) 000592 1781606646.121595 write 0x010097 thru 0x01917a (0x90e4 bytes) 000593 1781606646.123860 mapread 0x0378c1 thru 0x03b40a (0x3b4a bytes) 000594 1781606646.134053 punch from 0x01edc5 to 0x02aebe (0xc0fa bytes) 000595 1781606646.141260 mapwrite 0x015da2 thru 0x0253c6 (0xf625 bytes) 000596 1781606646.183616 trunc from 0x040000 to 0x013400 (0x2cbff bytes) 000597 1781606646.200544 fallocate 0x03e503 thru 0x03ffff (0x1afd bytes) 000598 1781606646.210589 read 0x00cd0e thru 0x013400 (0x66f3 bytes) 000599 1781606646.211286 write 0x02af2f thru 0x02ee49 (0x3f1b bytes) 000600 1781606646.212559 trunc from 0x02ee4a to 0x01df6a (0x10edf bytes) 000601 1781606646.233332 mapread 0x004e55 thru 0x00c494 (0x7640 bytes) 000602 1781606646.243869 trunc from 0x01df6b to 0x0013c6 (0x1cba4 bytes) 000603 1781606646.265308 trunc from 0x001372 to 0x0013c6 (0x55 bytes) 000604 1781606646.271862 read 0x000bac thru 0x0013c6 (0x81b bytes) 000605 1781606646.277394 mapread 0x000906 thru 0x0013c6 (0xac1 bytes) 000606 1781606646.278484 mapwrite 0x03f701 thru 0x03ffff (0x8ff bytes) 000607 1781606646.318267 read 0x03538d thru 0x03ffff (0xac73 bytes) 000608 1781606646.323925 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe172f926 (stale) Mirror 4: 0xe172f926 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xba60b8da Mirror 5: 0xba60b8da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000609 1781606647.005506 read 0x02697d thru 0x02bc68 (0x52ec bytes) 000610 1781606647.044056 mapread 0x00f36c thru 0x012439 (0x30ce bytes) 000611 1781606647.101754 fallocate 0x01bf68 thru 0x01d761 (0x17fa bytes) 000612 1781606647.163782 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe172f926 (stale) Mirror 4: 0xe172f926 (stale) Mirror 5: 0xba60b8da (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xba60b8da Mirror 6: 0xba60b8da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000613 1781606648.128896 trunc from 0x040000 to 0x0153c9 (0x2ac36 bytes) 000614 1781606648.192198 mapread 0x00853a thru 0x010cd9 (0x87a0 bytes) 000615 1781606648.216035 mapwrite 0x00d1b8 thru 0x01369a (0x64e3 bytes) 000616 1781606648.243271 read 0x00c057 thru 0x01198e (0x5938 bytes) 000617 1781606648.244314 trunc from 0x011832 to 0x0153c9 (0x3b98 bytes) 000618 1781606648.249808 mapwrite 0x002547 thru 0x00ad71 (0x882b bytes) 000619 1781606648.272214 mapwrite 0x037816 thru 0x03ea57 (0x7242 bytes) 000621 1781606648.334555 trunc from 0x03ea58 to 0x012c22 (0x2be35 bytes) 000622 1781606648.364297 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12c23): Mirror 3: 0x54d3bea3 (stale) Mirror 5: 0x54d3bea3 (stale) Mirror 6: 0x54d3bea3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12c23): Mirror 3: 0x54d3bea3 (stale) Mirror 5: 0x54d3bea3 (stale) Mirror 6: 0x54d3bea3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12c23: skipped 000623 1781606648.750077 trunc from 0x0072d9 to 0x011721 (0xa449 bytes) 000624 1781606648.758498 trunc from 0x003126 to 0x00dffa (0xaed5 bytes) 000625 1781606648.764351 mapwrite 0x03b9d6 thru 0x03bf88 (0x5b3 bytes) 000626 1781606648.811338 write 0x03f47f thru 0x03ffff (0xb81 bytes) 000628 1781606648.821421 MIRROR_RESYNC 4 mirrors 000630 1781606650.103510 read 0x02a962 thru 0x034059 (0x96f8 bytes) 000631 1781606650.113852 read 0x02047e thru 0x0288b9 (0x843c bytes) 000632 1781606650.124420 mapread 0x00ab14 thru 0x00f7a2 (0x4c8f bytes) 000633 1781606650.125355 trunc from 0x019dcd to 0x023b0f (0x9d43 bytes) 000634 1781606650.153122 trunc from 0x040000 to 0x037fbc (0x8043 bytes) 000635 1781606650.172749 trunc from 0x037fbd to 0x02835a (0xfc62 bytes) 000636 1781606650.190024 mapread 0x008669 thru 0x00cddd (0x4775 bytes) 000637 1781606650.215578 trunc from 0x023d60 to 0x02835a (0x45fb bytes) 000638 1781606650.220052 punch from 0x0275ec to 0x02835a (0xd6f bytes) 000639 1781606650.224815 mapread 0x026d30 thru 0x02835a (0x162b bytes) 000640 1781606650.238751 mapwrite 0x03b036 thru 0x03cb3a (0x1b05 bytes) 000641 1781606650.277558 fallocate 0x033244 thru 0x03ffff (0xcdbc bytes) 000642 1781606650.283280 trunc from 0x040000 to 0x0337e3 (0xc81c bytes) 000643 1781606650.299783 punch from 0x012721 to 0x01df4b (0xb82b bytes) 000644 1781606650.305365 read 0x00b209 thru 0x00cb57 (0x194f bytes) 000645 1781606650.306014 punch from 0x0192b4 to 0x019579 (0x2c6 bytes) 000646 1781606650.310609 punch from 0x031071 to 0x0337e3 (0x2773 bytes) 000647 1781606650.315197 read 0x01280b thru 0x017182 (0x4978 bytes) 000648 1781606650.325433 trunc from 0x009d69 to 0x00b0be (0x1356 bytes) 000649 1781606650.330278 trunc from 0x00379d to 0x00660a (0x2e6e bytes) 000650 1781606650.335107 punch from 0x01910e to 0x01f301 (0x61f4 bytes) 000651 1781606650.343441 read 0x0087e7 thru 0x00a6d1 (0x1eeb bytes) 000652 1781606650.352432 trunc from 0x0030c2 to 0x0123cf (0xf30e bytes) 000653 1781606650.357204 trunc from 0x0155c1 to 0x01b403 (0x5e43 bytes) 000654 1781606650.361378 punch from 0x0279d7 to 0x0337e3 (0xbe0d bytes) 000655 1781606650.366041 mapread 0x02f348 thru 0x0337e3 (0x449c bytes) 000656 1781606650.388305 trunc from 0x0337e4 to 0x0207b8 (0x1302b bytes) 000657 1781606650.406748 mapwrite 0x00de68 thru 0x0199ad (0xbb46 bytes) 000658 1781606650.479438 fallocate 0x034f93 thru 0x036f54 (0x1fc2 bytes) 000659 1781606650.489350 MIRROR_RESYNC 4 mirrors 000660 1781606651.543292 write 0x00c46d thru 0x0128fc (0x6490 bytes) 000661 1781606651.594494 trunc from 0x004ea7 to 0x010fdd (0xc137 bytes) 000662 1781606651.615437 fallocate 0x036886 thru 0x03aa83 (0x41fe bytes) 000663 1781606651.623948 trunc from 0x03aa84 to 0x00f667 (0x2b41c bytes) 000664 1781606651.658598 fallocate 0x02808e thru 0x036851 (0xe7c4 bytes) 000665 1781606651.667340 mapread 0x00e403 thru 0x00f667 (0x1265 bytes) 000666 1781606651.682779 read 0x00e665 thru 0x00f667 (0x1003 bytes) 000667 1781606651.683857 read 0x00712b thru 0x009f7a (0x2e50 bytes) 000668 1781606651.696702 mapread 0x003446 thru 0x00f667 (0xc222 bytes) 000669 1781606651.698187 mapwrite 0x00b455 thru 0x015a30 (0xa5dc bytes) 000670 1781606651.747077 write 0x01a226 thru 0x021c67 (0x7a42 bytes) 000672 1781606651.751039 trunc from 0x021c68 to 0x00faec (0x1217b bytes) 000673 1781606651.766907 read 0x009dca thru 0x00faec (0x5d23 bytes) 000674 1781606651.768379 mapwrite 0x0173a7 thru 0x01ec68 (0x78c2 bytes) 000675 1781606651.820392 read 0x0177c8 thru 0x018083 (0x8bc bytes) 000676 1781606651.821369 fallocate 0x0111b2 thru 0x0114f5 (0x344 bytes) 000678 1781606651.827993 fallocate 0x034eb5 thru 0x035686 (0x7d2 bytes) 000679 1781606651.834121 punch from 0x00b396 to 0x0166af (0xb31a bytes) 000681 1781606651.840703 trunc from 0x035687 to 0x01da28 (0x17c5e bytes) 000682 1781606651.857868 fallocate 0x03471a thru 0x03ffff (0xb8e6 bytes) 000683 1781606651.863394 write 0x02dd4f thru 0x03c49a (0xe74c bytes) 000685 1781606651.866553 read 0x000728 thru 0x008606 (0x7edf bytes) 000686 1781606651.867993 fallocate 0x038ebd thru 0x03cbbc (0x3d00 bytes) 000688 1781606651.874375 write 0x01956e thru 0x0215bd (0x8050 bytes) 000689 1781606651.889400 mapread 0x01c209 thru 0x0225d7 (0x63cf bytes) 000690 1781606651.903899 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3cbbd): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbbd): Mirror 3: 0xea8107f9 (stale) Mirror 5: 0xea8107f9 (stale) Mirror 6: 0xea8107f9 (stale) CRC-32 checksum value for chunk [0, 0x3cbbd): Mirror 1: 0xf4f20dd8 Mirror 7: 0xf4f20dd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbbd: skipped 000691 1781606652.935368 fallocate 0x036336 thru 0x03a14c (0x3e17 bytes) 000692 1781606652.981841 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cbbd): Mirror 5: 0xea8107f9 (stale) Mirror 6: 0xea8107f9 (stale) Mirror 7: 0xf4f20dd8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbbd): Mirror 5: 0xea8107f9 (stale) Mirror 6: 0xea8107f9 (stale) Mirror 7: 0xf4f20dd8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbbd: skipped 000693 1781606653.316044 read 0x021f3c thru 0x0255a4 (0x3669 bytes) 000694 1781606653.328196 trunc from 0x03a3ed to 0x03cbbc (0x27d0 bytes) 000695 1781606653.333892 write 0x00620e thru 0x00c3b2 (0x61a5 bytes) 000696 1781606653.348943 read 0x02a8b9 thru 0x036667 (0xbdaf bytes) 000697 1781606653.359880 write 0x00a7e7 thru 0x016f67 (0xc781 bytes) 000698 1781606653.362972 trunc from 0x021a5e to 0x02e02a (0xc5cd bytes) 000699 1781606653.368370 trunc from 0x01d96c to 0x01da37 (0xcc bytes) 000700 1781606653.381905 fallocate 0x031dbd thru 0x03202e (0x272 bytes) 000701 1781606653.389992 write 0x036c5c thru 0x039ea7 (0x324c bytes) 000702 1781606653.394448 trunc from 0x020262 to 0x02b868 (0xb607 bytes) 000703 1781606653.408265 punch from 0x02ce98 to 0x02d9f6 (0xb5f bytes) 000704 1781606653.419414 trunc from 0x03cbbd to 0x02853d (0x1467f bytes) 000705 1781606653.447733 punch from 0x01ce3c to 0x027c73 (0xae38 bytes) 000706 1781606653.455478 read 0x023f51 thru 0x02853d (0x45ed bytes) 000707 1781606653.471959 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2853e): Mirror 6: 0xc6f24f3a (stale) Mirror 7: 0x22b84ba0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2853e): Mirror 6: 0xc6f24f3a (stale) Mirror 7: 0x22b84ba0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2853e: skipped 000708 1781606653.955573 read 0x01b884 thru 0x021655 (0x5dd2 bytes) 000709 1781606653.968716 mapwrite 0x0082dd thru 0x00cf62 (0x4c86 bytes) 000710 1781606654.013167 write 0x01cc10 thru 0x02a6e5 (0xdad6 bytes) 000711 1781606654.020901 mapwrite 0x013f37 thru 0x023e46 (0xff10 bytes) 000712 1781606654.055061 mapwrite 0x01e54a thru 0x02aac0 (0xc577 bytes) 000713 1781606654.094891 fallocate 0x00b7bb thru 0x012131 (0x6977 bytes) 000714 1781606654.100321 punch from 0x00bfde to 0x019c22 (0xdc45 bytes) 000715 1781606654.106392 punch from 0x002640 to 0x00cead (0xa86e bytes) 000716 1781606654.112074 read 0x01ec98 thru 0x0296b7 (0xaa20 bytes) 000717 1781606654.112826 fallocate 0x018ace thru 0x01f2b8 (0x67eb bytes) 000718 1781606654.118377 punch from 0x01d769 to 0x01e789 (0x1021 bytes) 000719 1781606654.124649 fallocate 0x01987a thru 0x02012f (0x68b6 bytes) 000720 1781606654.133309 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2aac1): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aac1): Mirror 6: 0x17266dbf (stale) Mirror 7: 0x8ade55d4 (stale) CRC-32 checksum value for chunk [0, 0x2aac1): Mirror 1: 0x8dc55d3a Mirror 8: 0x8dc55d3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aac1: skipped 000721 1781606654.927865 punch from 0x01238b to 0x020453 (0xe0c9 bytes) 000723 1781606654.977043 write 0x03b321 thru 0x03c91d (0x15fd bytes) 000724 1781606654.988924 MIRROR_RESYNC 4 mirrors 000725 1781606656.127848 mapwrite 0x02b798 thru 0x02d392 (0x1bfb bytes) 000726 1781606656.225039 mapwrite 0x039331 thru 0x03bbf4 (0x28c4 bytes) 000727 1781606656.253880 fallocate 0x021610 thru 0x0230c4 (0x1ab5 bytes) 000728 1781606656.259128 trunc from 0x0050d5 to 0x00bdfd (0x6d29 bytes) 000729 1781606656.264904 trunc from 0x01a198 to 0x01da20 (0x3889 bytes) 000730 1781606656.270407 punch from 0x006031 to 0x008279 (0x2249 bytes) 000731 1781606656.275746 read 0x031158 thru 0x03c91d (0xb7c6 bytes) 000732 1781606656.276702 trunc from 0x02838a to 0x0308f7 (0x856e bytes) 000733 1781606656.283355 mapread 0x031f5e thru 0x034409 (0x24ac bytes) 000734 1781606656.284809 mapwrite 0x01141b thru 0x01bba9 (0xa78f bytes) 000735 1781606656.355934 fallocate 0x005f4e thru 0x00e53c (0x85ef bytes) 000736 1781606656.361887 fallocate 0x01c7e8 thru 0x02ae63 (0xe67c bytes) 000737 1781606656.366046 read 0x006d24 thru 0x0166d5 (0xf9b2 bytes) 000738 1781606656.375969 read 0x038527 thru 0x03c91d (0x43f7 bytes) 000739 1781606656.376648 fallocate 0x03562d thru 0x03ffff (0xa9d3 bytes) 000740 1781606656.382832 write 0x008a3f thru 0x00d69a (0x4c5c bytes) 000741 1781606656.392508 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c91e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c91e): Mirror 6: 0xf6f73210 (stale) Mirror 7: 0xf6f73210 (stale) Mirror 8: 0xf6f73210 (stale) CRC-32 checksum value for chunk [0, 0x3c91e): Mirror 1: 0x729600a3 Mirror 9: 0x729600a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c91e: skipped 000742 1781606657.457616 trunc from 0x03c91e to 0x039df5 (0x2b28 bytes) 000743 1781606657.533760 trunc from 0x039df6 to 0x00009a (0x39d5b bytes) 000745 1781606657.551694 trunc from 0x00009b to 0x00de1e (0xdd84 bytes) 000746 1781606657.572672 read 0x0037cf thru 0x0039d7 (0x209 bytes) 000747 1781606657.584646 trunc from 0x000a43 to 0x0032d5 (0x2893 bytes) 000748 1781606657.592813 trunc from 0x00de1f to 0x016b18 (0x8cfa bytes) 000749 1781606657.633362 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16b19): Mirror 6: 0x68d2c7ec (stale) Mirror 7: 0x68d2c7ec (stale) Mirror 8: 0x68d2c7ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16b19): Mirror 6: 0x68d2c7ec (stale) Mirror 7: 0x68d2c7ec (stale) Mirror 8: 0x68d2c7ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16b19: skipped 000750 1781606658.049485 trunc from 0x0061b4 to 0x011dda (0xbc27 bytes) 000751 1781606658.055070 mapwrite 0x01d8d7 thru 0x026b56 (0x9280 bytes) 000752 1781606658.158571 mapwrite 0x02b2ce thru 0x02de83 (0x2bb6 bytes) 000753 1781606658.206904 punch from 0x000ee3 to 0x00d045 (0xc163 bytes) 000755 1781606658.214977 trunc from 0x00a08e to 0x00ba26 (0x1999 bytes) 000756 1781606658.220080 write 0x02aec8 thru 0x03429b (0x93d4 bytes) 000757 1781606658.222608 mapwrite 0x03ac4c thru 0x03ffff (0x53b4 bytes) 000758 1781606658.299476 write 0x03607c thru 0x03ffff (0x9f84 bytes) 000759 1781606658.303793 mapread 0x0337a5 thru 0x03575c (0x1fb8 bytes) 000760 1781606658.307022 punch from 0x02f5f0 to 0x036caa (0x76bb bytes) 000761 1781606658.330283 mapread 0x02e350 thru 0x03d830 (0xf4e1 bytes) 000762 1781606658.342564 trunc from 0x00447c to 0x00b495 (0x701a bytes) 000763 1781606658.347943 mapwrite 0x0133f1 thru 0x0216a8 (0xe2b8 bytes) 000764 1781606658.384870 write 0x00a0db thru 0x00deaa (0x3dd0 bytes) 000765 1781606658.391911 trunc from 0x01fdbe to 0x021bf8 (0x1e3b bytes) 000766 1781606658.404964 write 0x0188f3 thru 0x01adbf (0x24cd bytes) 000767 1781606658.406545 punch from 0x034ed5 to 0x038dab (0x3ed7 bytes) 000768 1781606658.413737 mapread 0x0281ff thru 0x02a5ea (0x23ec bytes) 000769 1781606658.415062 mapwrite 0x03caa4 thru 0x03ffff (0x355c bytes) 000770 1781606658.459773 mapwrite 0x018fd1 thru 0x0243b2 (0xb3e2 bytes) 000771 1781606658.535942 trunc from 0x00b7c2 to 0x0116fe (0x5f3d bytes) 000772 1781606658.550045 write 0x01148c thru 0x014301 (0x2e76 bytes) 000773 1781606658.559590 write 0x00d6bd thru 0x010ef6 (0x383a bytes) 000774 1781606658.576209 write 0x002774 thru 0x00513c (0x29c9 bytes) 000775 1781606658.591372 fallocate 0x011880 thru 0x020e42 (0xf5c3 bytes) 000776 1781606658.601969 mapwrite 0x03dbcc thru 0x03ffff (0x2434 bytes) 000777 1781606658.652597 read 0x00e268 thru 0x01afbb (0xcd54 bytes) 000778 1781606658.653800 mapwrite 0x0278f5 thru 0x033ce9 (0xc3f5 bytes) 000779 1781606658.692474 trunc from 0x040000 to 0x022a1d (0x1d5e2 bytes) 000780 1781606658.707300 read 0x00751f thru 0x010d3c (0x981e bytes) 000781 1781606658.719270 punch from 0x002e71 to 0x00b5e7 (0x8777 bytes) 000782 1781606658.730369 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22a1e): Mirror 6: 0x7b3dd70e (stale) Mirror 7: 0x7b3dd70e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22a1e): Mirror 6: 0x7b3dd70e (stale) Mirror 7: 0x7b3dd70e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22a1e: skipped 000783 1781606659.114820 punch from 0x011a57 to 0x016d5f (0x5309 bytes) 000785 1781606659.123912 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22a1e): Mirror 6: 0x7b3dd70e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22a1e): Mirror 6: 0x7b3dd70e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22a1e: skipped 000786 1781606659.374277 trunc from 0x003190 to 0x01068f (0xd500 bytes) 000787 1781606659.380917 read 0x01b618 thru 0x022a1d (0x7406 bytes) 000788 1781606659.388406 read 0x0183dc thru 0x01fbc4 (0x77e9 bytes) 000789 1781606659.398828 punch from 0x01e2c3 to 0x022a1d (0x475b bytes) 000790 1781606659.402289 mapwrite 0x01c727 thru 0x02ad51 (0xe62b bytes) 000791 1781606659.460667 mapread 0x019bbe thru 0x02943f (0xf882 bytes) 000792 1781606659.462381 trunc from 0x022106 to 0x0238cf (0x17ca bytes) 000793 1781606659.468008 read 0x011fde thru 0x01ddcd (0xbdf0 bytes) 000794 1781606659.472312 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) 000795 1781606659.608546 write 0x010c95 thru 0x01e338 (0xd6a4 bytes) 000796 1781606659.631034 mapread 0x028efb thru 0x02ad51 (0x1e57 bytes) 000797 1781606659.653301 read 0x029e17 thru 0x02ad51 (0xf3b bytes) 000798 1781606659.653883 punch from 0x00c152 to 0x015e7f (0x9d2e bytes) 000799 1781606659.667620 trunc from 0x01ed47 to 0x02ad51 (0xc00b bytes) 000800 1781606659.679530 read 0x021eee thru 0x02a166 (0x8279 bytes) 000801 1781606659.700539 trunc from 0x02ad52 to 0x01e382 (0xc9cf bytes) 000802 1781606659.720842 read 0x005f16 thru 0x015172 (0xf25d bytes) 000804 1781606659.722929 read 0x0144f4 thru 0x01e382 (0x9e8f bytes) 000806 1781606659.726638 fallocate 0x02b704 thru 0x036644 (0xaf41 bytes) 000807 1781606659.734782 trunc from 0x0290b5 to 0x031615 (0x8561 bytes) 000808 1781606659.740714 mapread 0x02c8a2 thru 0x036644 (0x9da3 bytes) 000809 1781606659.755250 trunc from 0x036645 to 0x01a7a9 (0x1be9b bytes) 000810 1781606659.771324 read 0x00e4b5 thru 0x012f7b (0x4ac7 bytes) 000811 1781606659.775848 MIRROR_EXTEND to 2 mirrors 000812 1781606660.459094 fallocate 0x0066dc thru 0x00ab69 (0x448e bytes) 000813 1781606660.503861 fallocate 0x006842 thru 0x00728b (0xa4a bytes) 000814 1781606660.512242 punch from 0x019cfc to 0x01a7a9 (0xaae bytes) 000815 1781606660.523692 write 0x02793b thru 0x02d4ae (0x5b74 bytes) 000816 1781606660.530522 mapwrite 0x00ad87 thru 0x00dd58 (0x2fd2 bytes) 000817 1781606660.578028 mapwrite 0x0146f6 thru 0x01f0dc (0xa9e7 bytes) 000818 1781606660.628491 trunc from 0x02bd6a to 0x02d4ae (0x1745 bytes) 000819 1781606660.632824 trunc from 0x029c35 to 0x02b87a (0x1c46 bytes) 000820 1781606660.637776 trunc from 0x02d4af to 0x00ee3c (0x1e672 bytes) 000823 1781606660.670510 mapread 0x007a07 thru 0x00dfeb (0x65e5 bytes) 000824 1781606660.674886 read 0x00d24d thru 0x00ee3c (0x1bf0 bytes) 000825 1781606660.679676 MIRROR_RESYNC 2 mirrors 000826 1781606661.259907 fallocate 0x0118ef thru 0x020c12 (0xf324 bytes) 000827 1781606661.300053 fallocate 0x025d12 thru 0x02fe91 (0xa180 bytes) 000828 1781606661.306752 fallocate 0x0133ed thru 0x01a13f (0x6d53 bytes) 000829 1781606661.312943 mapwrite 0x00c394 thru 0x011d54 (0x59c1 bytes) 000830 1781606661.370009 trunc from 0x011d55 to 0x00eb25 (0x322f bytes) 000831 1781606661.388863 punch from 0x000a5e to 0x00a875 (0x9e18 bytes) 000832 1781606661.395399 read 0x006f66 thru 0x00eb25 (0x7bc0 bytes) 000833 1781606661.404026 fallocate 0x028f4e thru 0x03192e (0x89e1 bytes) 000835 1781606661.411128 read 0x021f20 thru 0x02f26a (0xd34b bytes) 000836 1781606661.420806 mapread 0x01a084 thru 0x01c5cf (0x254c bytes) 000837 1781606661.422014 write 0x0267fc thru 0x02bdbe (0x55c3 bytes) 000838 1781606661.423838 trunc from 0x00c63c to 0x011e08 (0x57cd bytes) 000839 1781606661.428558 trunc from 0x03192f to 0x03e47c (0xcb4e bytes) 000840 1781606661.446735 read 0x02682e thru 0x029108 (0x28db bytes) 000841 1781606661.447592 punch from 0x00cfe7 to 0x0111d2 (0x41ec bytes) 000842 1781606661.451885 punch from 0x02cfcd to 0x03051b (0x354f bytes) 000843 1781606661.455756 read 0x03ae78 thru 0x03e47c (0x3605 bytes) 000844 1781606661.464685 mapread 0x01366c thru 0x01ba09 (0x839e bytes) 000845 1781606661.466381 mapwrite 0x00ecbc thru 0x012fec (0x4331 bytes) 000846 1781606661.491631 read 0x004f3b thru 0x00a2ec (0x53b2 bytes) 000847 1781606661.492807 punch from 0x01f8ca to 0x02da32 (0xe169 bytes) 000848 1781606661.497944 trunc from 0x03e47d to 0x01d512 (0x20f6a bytes) 000849 1781606661.518203 fallocate 0x006279 thru 0x0091bc (0x2f44 bytes) 000850 1781606661.525099 read 0x00e00b thru 0x01cc82 (0xec78 bytes) 000853 1781606661.530037 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) 000854 1781606661.715028 mapread 0x00c270 thru 0x0140b8 (0x7e49 bytes) 000855 1781606661.761098 mapwrite 0x006551 thru 0x0098e2 (0x3392 bytes) 000856 1781606661.802384 trunc from 0x01cdbf to 0x01d512 (0x754 bytes) 000857 1781606661.812623 punch from 0x018ee0 to 0x01d512 (0x4633 bytes) 000858 1781606661.823926 write 0x01799a thru 0x0233eb (0xba52 bytes) 000859 1781606661.852746 trunc from 0x01ceaf to 0x01e100 (0x1252 bytes) 000862 1781606661.915670 mapwrite 0x026414 thru 0x03446a (0xe057 bytes) 000863 1781606662.018911 trunc from 0x01ee37 to 0x026e31 (0x7ffb bytes) 000864 1781606662.030313 trunc from 0x03446b to 0x003a35 (0x30a35 bytes) 000865 1781606662.061473 MIRROR_EXTEND to 2 mirrors 000866 1781606662.741934 trunc from 0x003219 to 0x003a35 (0x81d bytes) 000867 1781606662.790024 write 0x01d550 thru 0x0224b9 (0x4f6a bytes) 000868 1781606662.796830 punch from 0x0181cd to 0x01e510 (0x6344 bytes) 000869 1781606662.817551 trunc from 0x0224ba to 0x0255a8 (0x30ef bytes) 000870 1781606662.841281 mapwrite 0x02fa04 thru 0x035e41 (0x643e bytes) 000871 1781606662.916717 write 0x004e6a thru 0x0130c9 (0xe260 bytes) 000872 1781606662.925327 read 0x0090ba thru 0x00cca6 (0x3bed bytes) 000873 1781606662.926036 write 0x02c152 thru 0x037930 (0xb7df bytes) 000874 1781606662.931281 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) 000875 1781606663.037483 fallocate 0x03e6af thru 0x03f66f (0xfc1 bytes) 000876 1781606663.043949 punch from 0x00c255 to 0x01b598 (0xf344 bytes) 000877 1781606663.054820 trunc from 0x037931 to 0x00e7f3 (0x2913d bytes) 000878 1781606663.088588 punch from 0x00cadf to 0x00e7f3 (0x1d15 bytes) 000880 1781606663.095949 read 0x00629d thru 0x00d42f (0x7193 bytes) 000882 1781606663.105987 fallocate 0x01dd39 thru 0x02137e (0x3646 bytes) 000883 1781606663.112862 trunc from 0x00e7f4 to 0x02d775 (0x1ef82 bytes) 000884 1781606663.138708 fallocate 0x0337d7 thru 0x03ffff (0xc829 bytes) 000885 1781606663.146721 mapwrite 0x00cc82 thru 0x00f671 (0x29f0 bytes) 000886 1781606663.186750 fallocate 0x000e7b thru 0x002488 (0x160e bytes) 000887 1781606663.191733 punch from 0x01fccd to 0x02d775 (0xdaa9 bytes) 000888 1781606663.197519 fallocate 0x017a66 thru 0x01a244 (0x27df bytes) 000889 1781606663.205704 trunc from 0x02d776 to 0x03c4e8 (0xed73 bytes) 000890 1781606663.244775 mapwrite 0x018235 thru 0x02044b (0x8217 bytes) 000891 1781606663.288696 mapread 0x00695b thru 0x014ea7 (0xe54d bytes) 000892 1781606663.299930 trunc from 0x0167aa to 0x01dd4c (0x75a3 bytes) 000893 1781606663.311442 mapread 0x039992 thru 0x03c4e8 (0x2b57 bytes) 000895 1781606663.321321 fallocate 0x00b0bd thru 0x01925f (0xe1a3 bytes) 000896 1781606663.331359 mapwrite 0x03d122 thru 0x03ffff (0x2ede bytes) 000897 1781606663.386245 trunc from 0x00809e to 0x016aee (0xea51 bytes) 000898 1781606663.398277 mapwrite 0x01352c thru 0x0232ed (0xfdc2 bytes) 000899 1781606663.479331 mapread 0x035afa thru 0x035b4b (0x52 bytes) 000900 1781606663.482737 trunc from 0x040000 to 0x035213 (0xadec bytes) 000901 1781606663.514862 punch from 0x0044ad to 0x00ad32 (0x6886 bytes) 000902 1781606663.524112 punch from 0x027d47 to 0x03018e (0x8448 bytes) 000903 1781606663.530133 mapwrite 0x026392 thru 0x02fab5 (0x9724 bytes) 000904 1781606663.568755 trunc from 0x01f725 to 0x023b5a (0x4436 bytes) 000906 1781606663.574625 mapread 0x02aef8 thru 0x02b8d6 (0x9df bytes) 000907 1781606663.579302 MIRROR_EXTEND to 2 mirrors 000908 1781606664.392343 punch from 0x03116a to 0x035213 (0x40aa bytes) 000909 1781606664.424540 mapread 0x012c1b thru 0x018dd4 (0x61ba bytes) 000910 1781606664.444113 read 0x01c980 thru 0x0296f8 (0xcd79 bytes) 000911 1781606664.450669 read 0x00ea26 thru 0x00f109 (0x6e4 bytes) 000912 1781606664.451268 mapwrite 0x0342e5 thru 0x03c70d (0x8429 bytes) 000913 1781606664.489301 fallocate 0x035139 thru 0x03ffff (0xaec7 bytes) 000914 1781606664.502381 fallocate 0x02f1f1 thru 0x03e4d5 (0xf2e5 bytes) 000915 1781606664.513004 trunc from 0x012939 to 0x015f29 (0x35f1 bytes) 000916 1781606664.519653 mapread 0x02c747 thru 0x02ed50 (0x260a bytes) 000917 1781606664.524057 mapwrite 0x01e439 thru 0x028f25 (0xaaed bytes) 000918 1781606664.548288 punch from 0x037f9c to 0x03ffff (0x8064 bytes) 000919 1781606664.561342 write 0x001334 thru 0x0015a3 (0x270 bytes) 000921 1781606664.567824 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1815e6b9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf8ac7f15 Mirror 3: 0xf8ac7f15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000922 1781606665.618453 trunc from 0x040000 to 0x03cd77 (0x3288 bytes) 000923 1781606665.674201 trunc from 0x03cd78 to 0x005120 (0x37c57 bytes) 000924 1781606665.723008 fallocate 0x0220aa thru 0x028d65 (0x6cbc bytes) 000925 1781606665.740459 trunc from 0x005121 to 0x03d2ec (0x381cc bytes) 000926 1781606665.763789 fallocate 0x020311 thru 0x02456b (0x425b bytes) 000927 1781606665.770584 write 0x010e71 thru 0x013bda (0x2d6a bytes) 000929 1781606665.786709 mapwrite 0x00b199 thru 0x01a249 (0xf0b1 bytes) 000930 1781606665.830766 trunc from 0x03d2ed to 0x021417 (0x1bed5 bytes) 000931 1781606665.858472 trunc from 0x021418 to 0x019fd1 (0x7446 bytes) 000932 1781606665.894643 MIRROR_RESYNC 3 mirrors 000933 1781606666.849230 write 0x01ec8e thru 0x02bd45 (0xd0b8 bytes) 000934 1781606666.873292 write 0x03ebb4 thru 0x03ffff (0x144c bytes) 000935 1781606666.875319 write 0x02f068 thru 0x03a8de (0xb877 bytes) 000936 1781606666.886101 trunc from 0x028c94 to 0x037550 (0xe8bd bytes) 000937 1781606666.902010 trunc from 0x024059 to 0x02c049 (0x7ff1 bytes) 000938 1781606666.908003 trunc from 0x02ec71 to 0x03b249 (0xc5d9 bytes) 000939 1781606666.913235 mapwrite 0x008adb thru 0x013d07 (0xb22d bytes) 000940 1781606666.983634 mapwrite 0x01228a thru 0x01ffdc (0xdd53 bytes) 000941 1781606667.023814 punch from 0x010ca3 to 0x011d6d (0x10cb bytes) 000942 1781606667.035540 MIRROR_RESYNC 3 mirrors 000944 1781606668.075913 fallocate 0x01e316 thru 0x02a77d (0xc468 bytes) 000945 1781606668.160112 mapwrite 0x00d057 thru 0x013669 (0x6613 bytes) 000947 1781606668.219763 mapwrite 0x01b069 thru 0x020758 (0x56f0 bytes) 000948 1781606668.266231 punch from 0x024f16 to 0x026e8b (0x1f76 bytes) 000950 1781606668.274937 trunc from 0x02bd90 to 0x02cfad (0x121e bytes) 000951 1781606668.284464 mapread 0x02b244 thru 0x02bf27 (0xce4 bytes) 000952 1781606668.309074 punch from 0x02881e to 0x032359 (0x9b3c bytes) 000955 1781606668.318057 fallocate 0x03bc09 thru 0x03c9d2 (0xdca bytes) 000956 1781606668.332104 MIRROR_RESYNC 3 mirrors 000957 1781606669.230480 read 0x0264f5 thru 0x0342fd (0xde09 bytes) 000958 1781606669.247994 MIRROR_EXTEND to 4 mirrors 000959 1781606670.237046 mapwrite 0x00fba9 thru 0x01a80f (0xac67 bytes) 000960 1781606670.350492 trunc from 0x040000 to 0x035e73 (0xa18c bytes) 000962 1781606670.407679 trunc from 0x00204f to 0x00e856 (0xc808 bytes) 000963 1781606670.413244 fallocate 0x0004bc thru 0x004dc2 (0x4907 bytes) 000964 1781606670.417111 fallocate 0x02c34f thru 0x02ce9b (0xb4d bytes) 000965 1781606670.421016 fallocate 0x02a096 thru 0x03a00d (0xff78 bytes) 000966 1781606670.424660 read 0x02c8ce thru 0x032cd0 (0x6403 bytes) 000967 1781606670.431466 trunc from 0x01eceb to 0x026c39 (0x7f4f bytes) 000968 1781606670.438795 trunc from 0x01e3d1 to 0x02d9d5 (0xf605 bytes) 000969 1781606670.444807 trunc from 0x03a005 to 0x03a00d (0x9 bytes) 000970 1781606670.454257 mapwrite 0x00bca1 thru 0x010f20 (0x5280 bytes) 000971 1781606670.504665 punch from 0x033e6d to 0x037ab4 (0x3c48 bytes) 000973 1781606670.510846 trunc from 0x03a00e to 0x003597 (0x36a76 bytes) 000974 1781606670.528506 trunc from 0x00260a to 0x003597 (0xf8e bytes) 000975 1781606670.534588 read 0x000514 thru 0x003597 (0x3084 bytes) 000977 1781606670.543534 write 0x02b7c9 thru 0x03b1ff (0xfa37 bytes) 000979 1781606670.551476 read 0x003deb thru 0x01021f (0xc435 bytes) 000980 1781606670.560378 read 0x021615 thru 0x02297d (0x1369 bytes) 000981 1781606670.561069 trunc from 0x033a87 to 0x03b1ff (0x7779 bytes) 000982 1781606670.576020 fallocate 0x012d32 thru 0x020634 (0xd903 bytes) 000983 1781606670.581680 write 0x03d162 thru 0x03daf7 (0x996 bytes) 000985 1781606670.583288 punch from 0x00716b to 0x009807 (0x269d bytes) 000986 1781606670.588800 punch from 0x00bee0 to 0x01ab75 (0xec96 bytes) 000987 1781606670.594216 trunc from 0x03daf8 to 0x030287 (0xd870 bytes) 000989 1781606670.612930 read 0x01fb5d thru 0x020097 (0x53b bytes) 000991 1781606670.614245 fallocate 0x03ff51 thru 0x03ffff (0xaf bytes) 000993 1781606670.623682 write 0x038711 thru 0x03fdab (0x769b bytes) 000994 1781606670.636036 read 0x00d6d7 thru 0x00f48d (0x1db7 bytes) 000995 1781606670.638093 fallocate 0x000634 thru 0x004437 (0x3e04 bytes) 000996 1781606670.644178 fallocate 0x015f18 thru 0x02161d (0xb706 bytes) 000997 1781606670.650085 read 0x03aa0d thru 0x03e517 (0x3b0b bytes) 000998 1781606670.651296 fallocate 0x0301d2 thru 0x03c55d (0xc38c bytes) 000999 1781606670.661390 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xefc20d (stale) Mirror 3: 0xefc20d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xefc20d (stale) Mirror 3: 0xefc20d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped All operations completed A-OK! PASS 70a (61s) == sanity-flr test 71: check mirror extend parent fid ==== 06:44:36 (1781606676) ** lfs fid2path /mnt/lustre 0x200000404:0x23a:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 06:44:45 (1781606685) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x38a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.282677 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x38a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921136 s, 11.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x38b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x38a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x38b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x38a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x36f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 06:44:52 (1781606692) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8511424 .. Extending file size to 7145536 .. failed Extending file size to 8916320 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8953312 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9193056 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9255456 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59127 59128 59129 59132 59134 done 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) 192.168.201.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:46:14 (1781606774) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6018720 .. Extending file size to 3129088 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7334976 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6899328 .. Extending file size to 7721376 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8568608 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8568608 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8613696 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8976768 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8759328 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8901664 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8920448 .. Waiting 60915 60916 60917 60919 60920 done 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) 192.168.201.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:47:31 (1781606851) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2364 1268940 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3833116 16464 3590556 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23252 3583768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39716 7174324 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2364 1268940 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3833116 16464 3590556 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23252 3583768 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 39716 7174324 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0999154 s, 21.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12501 s, 16.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x33a36bc2 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) PASS 200c (14s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:47:46 (1781606866) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0492565 s, 42.6 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 06:47:52 (1781606872) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x249:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a2:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x384:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:47:59 (1781606879) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11556 s, 18.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x385:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x386:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:48:05 (1781606885) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124504 s, 16.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x387:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x387:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x388:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:48:12 (1781606892) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0710771 s, 29.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x389:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x389:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x38a:0x0] } PASS 204c (31s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:48:44 (1781606924) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg123-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818878 s, 12.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x38c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x38b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x38c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x38b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a8:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:49:31 (1781606971) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg123-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3aa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109022 s, 19.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3aa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x38d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3aa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x38d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:50:19 (1781607019) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg123-server mds-ost sync done. Creating new pool test_204f oleg123-server: Pool lustre.test_204f created Adding targets to pool oleg123-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg123-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.125955 s, 16.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x38e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x3ab:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x38e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg123-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg123-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg123-server: Pool lustre.test_204f destroyed PASS 204f (65s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:51:25 (1781607085) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:51:31 (1781607091) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:51:37 (1781607097) oleg123-server: Pool lustre.test_206 created Adding targets to pool oleg123-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg123-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg123-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg123-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg123-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:51:59 (1781607119) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464881 s, 22.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366059 s, 28.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x898a6696 Mirror 3: 0x898a6696 PASS 207 (8s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:52:07 (1781607127) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:52:10 (1781607130) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:52:13 (1781607133) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0451657 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x13b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1960000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1960000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1960000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2370000 Read this (4 bytes): BBBB 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:52:41 (1781607161) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0151971 s, 539 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:52:47 (1781607167) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058772 s, 17.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x26e lmm_fid: [0x200000404:0x26e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x26e lmm_fid: [0x200000404:0x26e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x398:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834542 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 06:52:48.000000000 -0400 Modify: 2026-06-16 06:52:48.000000000 -0400 Change: 2026-06-16 06:52:49.000000000 -0400 Birth: 2026-06-16 06:52:48.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:52:53 (1781607173) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148788 s, 70.5 MB/s oleg123-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg123-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg123-server: O/240000400/d22/3b6: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:53:14 (1781607194) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.193623 s, 212 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0223103 s, 184 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 16 06:53 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x273:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 16 06:53 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:53:20 (1781607200) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:53:25 (1781607205) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1731 sec =========== 06:53:31 (1781607211) === sanity-flr: start cleanup 06:53:32 (1781607212) === === sanity-flr: finish cleanup 06:53:37 (1781607217) ===