-----============= acceptance-small: sanity-flr ============----- Fri Sep 4 07:36:08 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg616-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 07:36:25 (1788521785) === oleg616-client.virtnet: executing check_config_client /mnt/lustre oleg616-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg616-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9281d044f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9281d044f800.idle_timeout=debug disable quota as required oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 07:36:53 (1788521813) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7619] == sanity-flr test 0a: lfs mirror create with -N option == 07:36:55 (1788521815) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:37:11 (1788521831) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:37:15 (1788521835) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:37:19 (1788521839) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:37:43 (1788521863) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:37:46 (1788521866) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 07:37:50 (1788521870) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9281d044f800.stats=clear osc.lustre-OST0001-osc-ffff9281d044f800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:37:58 (1788521878) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:38:10 (1788521890) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:38:19 (1788521899) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:38:22 (1788521902) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (6s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:38:28 (1788521908) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 07:38:36 (1788521916) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:38:43 (1788521923) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:38:46 (1788521926) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:38:54 (1788521934) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:39:04 (1788521944) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:39:11 (1788521951) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.23512 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14381 s, 7.3 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:39:24 (1788521964) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.136971 s, 30.6 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:39:32 (1788521972) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:39:42 (1788521982) Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9281d044f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9281d044f800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9281d044f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9281d044f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg616-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:40:55 (1788522055) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.33527 s, 18.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9281d044f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9281d044f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:41:35 (1788522095) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9281d044f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9281d044f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9281d044f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9281d044f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg616-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:42:56 (1788522176) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.93688 s, 8.8 MB/s -rw-r--r-- 1 root root 52428800 Sep 4 07:43 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.177s user 0m0.001s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9281d044f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9281d044f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.102s user 0m0.001s sys 0m0.198s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9281d044f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9281d044f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.404s user 0m0.001s sys 0m0.174s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg616-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg616-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9281d044f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9281d044f800.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:44:28 (1788522268) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:44:32 (1788522272) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:44:36 (1788522276) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 07:44:40 (1788522280) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 07:44:47 (1788522287) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.704694 s, 17.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:45:01 (1788522301) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035866 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341869 s, 30.7 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:45:08 (1788522308) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:45:16 (1788522316) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00150769 s, 679 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00151033 s, 678 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00110757 s, 925 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00126899 s, 807 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 07:45:27 (1788522327) create test files with size 15743687 .. 5a287a84f202a2d07663620ac19f557f - 1fe68c4b7277663b301ef695cc883ba7 - c0bc621dccbcfdbe0f177b2b6fb71f42 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg616-client.virtnet /mnt/lustre (opts:) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 07:45:54 (1788522354) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:46:08 (1788522368) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:46:15 (1788522375) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119046 s, 8.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338179 s, 31.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 07:46:24 (1788522384) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136688 s, 15.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.375151 s, 11.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151723 s, 13.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 07:46:41 (1788522401) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 07:46:45 (1788522405) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:46:48 (1788522408) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264813 s, 39.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 07:46:56 (1788522416) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:47:00 (1788522420) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:47:11 (1788522431) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.286417 s, 36.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg616-client.virtnet /mnt/lustre (opts:) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:47:23 (1788522443) llite.lustre-ffff9281d044b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.289308 s, 36.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg616-client.virtnet /mnt/lustre (opts:) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9281c8c92000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:47:38 (1788522458) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:48:22 (1788522502) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:48:56 (1788522536) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 07:48:59 (1788522539) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.313542 s, 10.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.256221 s, 12.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc228a538 (nosync,stale) Mirror 2: 0xc228a538 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1b88d21c303e9ba2528dca10d510c668 - mirror 1 checksum: a6a23b0e8b02e5ff1c99f6cf8b358906 - mirror 2 checksum: 1b88d21c303e9ba2528dca10d510c668 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T07:49:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-04T07:49:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1b88d21c303e9ba2528dca10d510c668 - mirror 1 checksum: a6a23b0e8b02e5ff1c99f6cf8b358906 - mirror 2 checksum: a6a23b0e8b02e5ff1c99f6cf8b358906 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T07:49:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-04T07:49:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:49:12 (1788522552) Failing mds1 on oleg616-server Stopping /mnt/lustre-mds1 (opts:) on oleg616-server 07:49:19 (1788522559) shut down facet: mds1 facet_host: oleg616-server facet_failover_host: oleg616-server Failover mds1 to oleg616-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-MDT0000 07:49:50 (1788522590) targets are mounted 07:49:50 (1788522590) facet_failover done oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434527 s, 24.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 50A (59s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:50:11 (1788522611) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:50:15 (1788522615) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:50:18 (1788522618) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00714391 s, 2.3 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:50:25 (1788522625) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:50:28 (1788522628) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0114675 s, 357 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0811296 s, 50.5 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:50:36 (1788522636) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:51:03 (1788522663) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:51:32 (1788522692) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:51:55 (1788522715) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:52:20 (1788522740) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 07:52:29 (1788522749) mirror create pid 54291 mirror split pid 54293 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54291 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54293 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 07:53:35 (1788522815) Seed set to 2817 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x245c8 000007 1788522817.705929 trunc from 00000000 to 0x0245c7 (0x245c8 bytes) 000008 1788522817.728020 punch from 0x001f38 to 0x0086df (0x67a8 bytes) 000014 1788522817.738338 mapread 0x00279e thru 0x005275 (0x2ad8 bytes) 000015 1788522817.763078 mapread 0x00cffe thru 0x017ee5 (0xaee8 bytes) 000018 1788522817.779583 mapwrite 0x018454 thru 0x01c9fb (0x45a8 bytes) 000019 1788522817.920358 mapread 0x015487 thru 0x018496 (0x3010 bytes) 000020 1788522817.924877 mapwrite 0x01b5d8 thru 0x021eee (0x6917 bytes) 000021 1788522818.098998 punch from 0x0072cb to 0x00fa98 (0x87ce bytes) 000022 1788522818.109277 punch from 0x01ed33 to 0x0245c7 (0x5895 bytes) 000023 1788522818.119107 punch from 0x00eba5 to 0x0150e0 (0x653c bytes) 000024 1788522818.134114 mapread 0x007b91 thru 0x01438c (0xc7fc bytes) 000025 1788522818.176766 punch from 0x001db3 to 0x004d41 (0x2f8f bytes) 000026 1788522818.182542 read 0x02174a thru 0x0245c7 (0x2e7e bytes) 000027 1788522818.191027 read 0x022c9a thru 0x0245c7 (0x192e bytes) 000029 1788522818.192198 read 0x01c6da thru 0x02027d (0x3ba4 bytes) 000030 1788522818.215587 mapread 0x022819 thru 0x0245c7 (0x1daf bytes) 000032 1788522818.224026 MIRROR_EXTEND to 2 mirrors 000033 1788522819.423805 read 0x011c85 thru 0x012413 (0x78f bytes) 000034 1788522819.452477 MIRROR_EXTEND to 3 mirrors 000038 1788522820.550664 mapread 0x005efc thru 0x01444d (0xe552 bytes) 000039 1788522820.622158 read 0x023c1e thru 0x0245c7 (0x9aa bytes) 000040 1788522820.637035 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000041 1788522821.258331 mapwrite 0x00a412 thru 0x013bc1 (0x97b0 bytes) 000042 1788522821.657883 read 0x020d5a thru 0x0245c7 (0x386e bytes) 000043 1788522821.695312 write 0x032d64 thru 0x03ffff (0xd29c bytes) 000046 1788522821.704220 punch from 0x0172f4 to 0x01bda7 (0x4ab4 bytes) 000049 1788522821.740478 write 0x03026d thru 0x035da0 (0x5b34 bytes) 000050 1788522821.760352 punch from 0x01064e to 0x014d09 (0x46bc bytes) 000051 1788522821.791346 mapwrite 0x019358 thru 0x0196cf (0x378 bytes) 000052 1788522821.936267 read 0x0253be thru 0x026949 (0x158c bytes) 000055 1788522821.951995 punch from 0x01b463 to 0x02072b (0x52c9 bytes) 000056 1788522821.957418 punch from 0x00e33f to 0x01d142 (0xee04 bytes) 000057 1788522821.963100 mapwrite 0x02c907 thru 0x030655 (0x3d4f bytes) 000060 1788522822.007777 mapwrite 0x018632 thru 0x021a45 (0x9414 bytes) 000063 1788522822.095294 mapwrite 0x00b3df thru 0x00c0c6 (0xce8 bytes) 000064 1788522822.148068 write 0x026729 thru 0x029332 (0x2c0a bytes) 000066 1788522822.151090 read 0x01d9c3 thru 0x0230ae (0x56ec bytes) 000068 1788522822.158840 MIRROR_RESYNC 2 mirrors 000071 1788522823.370411 punch from 0x00ca3d to 0x01379f (0x6d63 bytes) 000072 1788522823.515477 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf99dfdd2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xee251a32 Mirror 3: 0xee251a32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000073 1788522824.888599 mapread 0x01afa7 thru 0x01fdfd (0x4e57 bytes) 000074 1788522824.941213 read 0x036b97 thru 0x03ffff (0x9469 bytes) 000075 1788522824.961884 read 0x034dcc thru 0x03ffff (0xb234 bytes) 000077 1788522824.994114 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf99dfdd2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xee251a32 Mirror 3: 0xee251a32 Mirror 4: 0xee251a32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x34eb1 000079 1788522826.348497 trunc from 0x040000 to 0x034eb0 (0xb14f bytes) 000080 1788522826.448119 mapread 0x009418 thru 0x00b0b9 (0x1ca2 bytes) 000081 1788522826.469959 trunc from 0x034eb1 to 0x02f7fc (0x56b4 bytes) 000082 1788522826.499364 read 0x005fb0 thru 0x006757 (0x7a8 bytes) 000084 1788522826.505340 write 0x0067f5 thru 0x00dc97 (0x74a3 bytes) 000086 1788522826.507261 mapread 0x01e25c thru 0x02d7ec (0xf591 bytes) 000087 1788522826.534942 punch from 0x01b07d to 0x0272c0 (0xc244 bytes) 000088 1788522826.557547 mapread 0x004f3b thru 0x00bff3 (0x70b9 bytes) 000089 1788522826.558877 trunc from 0x02f7fd to 0x00ceb8 (0x22944 bytes) 000090 1788522826.584761 mapwrite 0x027313 thru 0x033650 (0xc33e bytes) 000091 1788522826.703946 write 0x01c35b thru 0x023166 (0x6e0c bytes) 000092 1788522826.717169 mapwrite 0x03ba94 thru 0x03ffff (0x456c bytes) 000093 1788522826.860596 trunc from 0x040000 to 0x002d1c (0x3d2e3 bytes) 000094 1788522826.890015 write 0x033ea7 thru 0x03baab (0x7c05 bytes) 000096 1788522826.915198 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3baac): Mirror 3: 0x4c596557 (stale) Mirror 4: 0x4c596557 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baac): Mirror 3: 0x4c596557 (stale) Mirror 4: 0x4c596557 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baac: skipped 000098 1788522827.357918 mapread 0x006e50 thru 0x00fb0e (0x8cbf bytes) 000099 1788522827.422781 write 0x03e59f thru 0x03ffff (0x1a61 bytes) 000102 1788522827.426421 punch from 0x02a1b7 to 0x038824 (0xe66e bytes) 000103 1788522827.445695 trunc from 0x040000 to 0x0301c5 (0xfe3a bytes) 000104 1788522827.498233 punch from 0x012a7f to 0x019f7b (0x74fd bytes) 000105 1788522827.511190 mapread 0x019f33 thru 0x0202bf (0x638d bytes) 000106 1788522827.527147 punch from 0x00c38e to 0x0181d0 (0xbe43 bytes) 000107 1788522827.533618 punch from 0x017b1d to 0x021161 (0x9645 bytes) 000108 1788522827.549842 write 0x01e0e7 thru 0x02b5de (0xd4f8 bytes) 000110 1788522827.605296 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x301c6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301c6): Mirror 3: 0x57c8f485 (stale) Mirror 4: 0x57c8f485 (stale) CRC-32 checksum value for chunk [0, 0x301c6): Mirror 1: 0xdf66733d Mirror 5: 0xdf66733d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301c6: skipped 000111 1788522829.211031 punch from 0x02f05f to 0x0301c5 (0x1167 bytes) 000112 1788522829.456224 mapread 0x027e9d thru 0x02ac55 (0x2db9 bytes) 000113 1788522829.521768 read 0x016ced thru 0x01a8ec (0x3c00 bytes) 000114 1788522829.532932 write 0x0075f1 thru 0x017571 (0xff81 bytes) 000115 1788522829.541276 trunc from 0x0301c6 to 0x02c2f3 (0x3ed2 bytes) 000118 1788522829.582067 trunc from 0x02c2f4 to 0x002009 (0x2a2ea bytes) 000119 1788522829.600024 mapwrite 0x009250 thru 0x010e07 (0x7bb8 bytes) 000120 1788522829.693323 trunc from 0x010e08 to 0x025661 (0x1485a bytes) 000121 1788522829.722359 read 0x00a304 thru 0x017329 (0xd026 bytes) 000123 1788522829.745496 mapread 0x02312f thru 0x0245c4 (0x1496 bytes) 000124 1788522829.759987 mapwrite 0x039191 thru 0x03b6ae (0x251e bytes) 000126 1788522829.933602 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b6af): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6af): Mirror 3: 0x15c6e58 (stale) Mirror 4: 0x15c6e58 (stale) Mirror 5: 0x32b059b8 (stale) CRC-32 checksum value for chunk [0, 0x3b6af): Mirror 1: 0x85abae4b Mirror 6: 0x85abae4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6af: skipped 000128 1788522831.407012 read 0x029386 thru 0x02d268 (0x3ee3 bytes) 000129 1788522831.428597 write 0x02be06 thru 0x02e61b (0x2816 bytes) 000130 1788522831.524760 write 0x02da59 thru 0x03661d (0x8bc5 bytes) 000132 1788522831.529403 write 0x03bfd2 thru 0x03ffff (0x402e bytes) 000133 1788522831.535564 read 0x012fb6 thru 0x01e53f (0xb58a bytes) 000135 1788522831.547889 trunc from 0x040000 to 0x01efff (0x21000 bytes) 000136 1788522831.569398 mapread 0x01e9c6 thru 0x01edce (0x409 bytes) 000137 1788522831.570425 read 0x009f5e thru 0x011a27 (0x7aca bytes) 000138 1788522831.578263 mapwrite 0x03e776 thru 0x03ffff (0x188a bytes) 000139 1788522831.657112 write 0x009f79 thru 0x018079 (0xe101 bytes) 000140 1788522831.658589 mapread 0x001ac7 thru 0x009792 (0x7ccc bytes) 000142 1788522831.661190 write 0x00ab7a thru 0x01a799 (0xfc20 bytes) 000143 1788522831.663719 read 0x03f2a8 thru 0x03ffff (0xd58 bytes) 000146 1788522831.665888 read 0x01bd9f thru 0x024a41 (0x8ca3 bytes) 000148 1788522831.667766 mapwrite 0x01c08c thru 0x02bebe (0xfe33 bytes) 000149 1788522831.715750 write 0x0339ef thru 0x03aa11 (0x7023 bytes) 000150 1788522831.717705 write 0x02b153 thru 0x02ca56 (0x1904 bytes) 000151 1788522831.719236 write 0x002c02 thru 0x005457 (0x2856 bytes) 000152 1788522831.725801 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xee251a32 (stale) Mirror 4: 0xee251a32 (stale) Mirror 5: 0x134cb081 (stale) Mirror 6: 0xf215db4d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x703ba4a3 Mirror 7: 0x703ba4a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000154 1788522833.684039 read 0x0122ff thru 0x01706e (0x4d70 bytes) 000155 1788522833.730288 mapwrite 0x016277 thru 0x021b1c (0xb8a6 bytes) 000156 1788522834.044942 write 0x03c013 thru 0x03ffff (0x3fed bytes) 000157 1788522834.062291 trunc from 0x040000 to 0x022d8f (0x1d270 bytes) 000159 1788522834.095197 read 0x020b0a thru 0x022d8f (0x2286 bytes) 000160 1788522834.097700 trunc from 0x022d90 to 0x0035ba (0x1f7d5 bytes) 000161 1788522834.140769 mapread 0x002484 thru 0x0035ba (0x1137 bytes) 000162 1788522834.156863 punch from 0x0034bd to 0x0035ba (0xfe bytes) 000164 1788522834.162526 read 0x000820 thru 0x0035ba (0x2d9b bytes) 000167 1788522834.170205 trunc from 0x0035bb to 0x00a0c3 (0x6b09 bytes) 000169 1788522834.200102 punch from 0x004017 to 0x00a0c3 (0x60ad bytes) 000171 1788522834.207407 mapread 0x0077d4 thru 0x00a0c3 (0x28f0 bytes) 000173 1788522834.225372 mapread 0x006d61 thru 0x00a0c3 (0x3363 bytes) 000175 1788522834.233918 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xa0c4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa0c4): Mirror 3: 0xc4bd1462 (stale) Mirror 4: 0xc4bd1462 (stale) Mirror 5: 0xc4bd1462 (stale) Mirror 6: 0x6e810602 (stale) Mirror 7: 0x81875e50 (stale) CRC-32 checksum value for chunk [0, 0xa0c4): Mirror 1: 0x3b1a4581 Mirror 8: 0x3b1a4581 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa0c4: skipped 000176 1788522835.935512 mapread 0x007688 thru 0x009b10 (0x2489 bytes) 000177 1788522835.967606 punch from 0x00740c to 0x00a0c3 (0x2cb8 bytes) 000178 1788522836.046085 MIRROR_RESYNC 7 mirrors 000179 1788522838.376214 punch from 0x008f20 to 0x00a0c3 (0x11a4 bytes) 000180 1788522838.481796 punch from 0x009202 to 0x00a0c3 (0xec2 bytes) 000181 1788522838.486191 write 0x011f46 thru 0x01c0f3 (0xa1ae bytes) 000183 1788522838.489331 mapwrite 0x03b314 thru 0x03ffff (0x4cec bytes) 000184 1788522838.575483 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7fbec283 (stale) Mirror 4: 0x7fbec283 (stale) Mirror 5: 0x7fbec283 (stale) Mirror 6: 0x7fbec283 (stale) Mirror 7: 0x7fbec283 (stale) Mirror 8: 0x7fbec283 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x30d0c638 Mirror 9: 0x30d0c638 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000185 1788522839.853200 mapwrite 0x019996 thru 0x01cf17 (0x3582 bytes) 000186 1788522840.206259 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7fbec283 (stale) Mirror 5: 0x7fbec283 (stale) Mirror 6: 0x7fbec283 (stale) Mirror 7: 0x7fbec283 (stale) Mirror 8: 0x7fbec283 (stale) Mirror 9: 0x30d0c638 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7fbec283 (stale) Mirror 5: 0x7fbec283 (stale) Mirror 6: 0x7fbec283 (stale) Mirror 7: 0x7fbec283 (stale) Mirror 8: 0x7fbec283 (stale) Mirror 9: 0x30d0c638 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000189 1788522840.796996 MIRROR_RESYNC 7 mirrors 000190 1788522843.604978 MIRROR_RESYNC 7 mirrors 000191 1788522844.074746 write 0x004f3d thru 0x014c53 (0xfd17 bytes) 000192 1788522844.169157 MIRROR_RESYNC 7 mirrors 000194 1788522846.184157 trunc from 0x040000 to 0x01629f (0x29d60 bytes) 000195 1788522846.276462 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x162a0): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x162a0): Mirror 3: 0x43cc1cd2 (stale) Mirror 5: 0x43cc1cd2 (stale) Mirror 6: 0x43cc1cd2 (stale) Mirror 7: 0x43cc1cd2 (stale) Mirror 8: 0x43cc1cd2 (stale) Mirror 9: 0x43cc1cd2 (stale) CRC-32 checksum value for chunk [0, 0x162a0): Mirror 1: 0x43cc1cd2 Mirror 10: 0x43cc1cd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x162a0: skipped 000200 1788522847.267800 trunc from 0x0162a0 to 0x032a1f (0x1c780 bytes) 000201 1788522847.349122 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32a20): Mirror 5: 0x1b10f4f9 (stale) Mirror 6: 0x1b10f4f9 (stale) Mirror 7: 0x1b10f4f9 (stale) Mirror 8: 0x1b10f4f9 (stale) Mirror 9: 0x1b10f4f9 (stale) Mirror 10: 0xbabb245a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32a20): Mirror 5: 0x1b10f4f9 (stale) Mirror 6: 0x1b10f4f9 (stale) Mirror 7: 0x1b10f4f9 (stale) Mirror 8: 0x1b10f4f9 (stale) Mirror 9: 0x1b10f4f9 (stale) Mirror 10: 0xbabb245a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32a20: skipped 000202 1788522847.767990 write 0x030905 thru 0x039a2f (0x912b bytes) 000203 1788522847.778542 write 0x008f55 thru 0x00917d (0x229 bytes) truncating to largest ever: 0x3856a 000204 1788522847.794468 trunc from 0x039a30 to 0x038569 (0x14c6 bytes) 000206 1788522847.826846 mapread 0x032302 thru 0x036bf4 (0x48f3 bytes) 000210 1788522847.836413 MIRROR_RESYNC 7 mirrors 000211 1788522849.746278 read 0x0109a8 thru 0x01351d (0x2b76 bytes) 000212 1788522849.759104 punch from 0x021629 to 0x02ffe7 (0xe9bf bytes) 000214 1788522849.859726 mapwrite 0x029f39 thru 0x0399d2 (0xfa9a bytes) 000217 1788522849.993449 write 0x03fbd4 thru 0x03ffff (0x42c bytes) 000219 1788522849.996152 punch from 0x03ccf5 to 0x03ffff (0x330b bytes) 000220 1788522850.008610 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x89f7c22e (stale) Mirror 6: 0x89f7c22e (stale) Mirror 7: 0x89f7c22e (stale) Mirror 8: 0x89f7c22e (stale) Mirror 10: 0x89f7c22e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x89f7c22e (stale) Mirror 6: 0x89f7c22e (stale) Mirror 7: 0x89f7c22e (stale) Mirror 8: 0x89f7c22e (stale) Mirror 10: 0x89f7c22e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000222 1788522850.395926 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x89f7c22e (stale) Mirror 6: 0x89f7c22e (stale) Mirror 8: 0x89f7c22e (stale) Mirror 10: 0x89f7c22e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x89f7c22e (stale) Mirror 6: 0x89f7c22e (stale) Mirror 8: 0x89f7c22e (stale) Mirror 10: 0x89f7c22e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000223 1788522850.767457 punch from 0x01c796 to 0x029cd5 (0xd540 bytes) 000226 1788522850.778748 MIRROR_RESYNC 5 mirrors 000228 1788522852.174916 MIRROR_EXTEND to 6 mirrors 000230 1788522853.102440 punch from 0x039431 to 0x03ffff (0x6bcf bytes) 000231 1788522853.203929 trunc from 0x040000 to 0x00abd5 (0x3542a bytes) 000232 1788522853.249090 mapwrite 0x0292d4 thru 0x03698d (0xd6ba bytes) 000233 1788522853.393265 mapread 0x003619 thru 0x00dd5a (0xa742 bytes) 000234 1788522853.403215 read 0x009411 thru 0x018050 (0xec40 bytes) 000235 1788522853.414318 mapread 0x01dca9 thru 0x021b5a (0x3eb2 bytes) 000237 1788522853.425432 read 0x004fdb thru 0x00ee9e (0x9ec4 bytes) 000238 1788522853.426141 read 0x01f513 thru 0x022f56 (0x3a44 bytes) 000239 1788522853.431161 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3698e): Mirror 6: 0x7abaa3a0 (stale) Mirror 8: 0x7abaa3a0 (stale) Mirror 10: 0x7abaa3a0 (stale) Mirror 11: 0x7abaa3a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3698e): Mirror 6: 0x7abaa3a0 (stale) Mirror 8: 0x7abaa3a0 (stale) Mirror 10: 0x7abaa3a0 (stale) Mirror 11: 0x7abaa3a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3698e: skipped 000241 1788522853.789748 mapread 0x03556b thru 0x03698d (0x1423 bytes) 000242 1788522853.816051 punch from 0x015024 to 0x023fbb (0xef98 bytes) 000244 1788522853.829787 mapwrite 0x02c912 thru 0x02d641 (0xd30 bytes) 000245 1788522853.949085 trunc from 0x03698e to 0x00b64f (0x2b33e bytes) 000246 1788522853.974888 punch from 0x0050f2 to 0x009f2c (0x4e3b bytes) 000248 1788522853.989592 mapread 0x0084f2 thru 0x00b64f (0x315e bytes) 000250 1788522854.013051 read 0x0092ce thru 0x00b64f (0x2382 bytes) 000251 1788522854.013841 punch from 0x007b1b to 0x00b64f (0x3b35 bytes) 000252 1788522854.019895 punch from 0x009d2b to 0x00b64f (0x1925 bytes) truncating to largest ever: 0x3b489 000255 1788522854.038448 trunc from 0x00b650 to 0x03b488 (0x2fe39 bytes) 000256 1788522854.062611 mapwrite 0x03f5f3 thru 0x03ffff (0xa0d bytes) 000257 1788522854.143413 punch from 0x020f5f to 0x02739d (0x643f bytes) 000258 1788522854.157508 mapwrite 0x0290f9 thru 0x030222 (0x712a bytes) 000260 1788522854.229002 punch from 0x0209bd to 0x02fe6d (0xf4b1 bytes) 000261 1788522854.235492 trunc from 0x040000 to 0x039315 (0x6cea bytes) 000262 1788522854.254386 punch from 0x01a0de to 0x020c9f (0x6bc2 bytes) 000263 1788522854.260165 write 0x01ae58 thru 0x028f41 (0xe0ea bytes) 000265 1788522854.281440 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39316): Mirror 8: 0x67751e1 (stale) Mirror 10: 0x67751e1 (stale) Mirror 11: 0x67751e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39316): Mirror 8: 0x67751e1 (stale) Mirror 10: 0x67751e1 (stale) Mirror 11: 0x67751e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39316: skipped 000267 1788522854.745429 MIRROR_RESYNC 4 mirrors 000268 1788522856.086306 write 0x0084cb thru 0x017cbc (0xf7f2 bytes) 000270 1788522856.167479 read 0x008fe9 thru 0x016cee (0xdd06 bytes) 000271 1788522856.168207 trunc from 0x039316 to 0x01c4fb (0x1ce1a bytes) 000272 1788522856.194239 write 0x0204e5 thru 0x0248ce (0x43ea bytes) 000273 1788522856.201577 read 0x00ddb3 thru 0x017a28 (0x9c76 bytes) truncating to largest ever: 0x3e82e 000276 1788522856.204523 trunc from 0x0248cf to 0x03e82d (0x19f5f bytes) 000277 1788522856.222740 mapwrite 0x007718 thru 0x012e11 (0xb6fa bytes) 000278 1788522856.287272 mapwrite 0x03f672 thru 0x03ffff (0x98e bytes) 000280 1788522856.357911 write 0x03e0ff thru 0x03ffff (0x1f01 bytes) 000281 1788522856.360136 punch from 0x001220 to 0x0076a6 (0x6487 bytes) 000282 1788522856.372761 trunc from 0x040000 to 0x024fd0 (0x1b02f bytes) 000284 1788522856.406816 trunc from 0x024fd1 to 0x01ef47 (0x6089 bytes) 000285 1788522856.435275 MIRROR_RESYNC 4 mirrors 000289 1788522857.594927 read 0x00ecc3 thru 0x018993 (0x9cd1 bytes) 000290 1788522857.613368 read 0x010e04 thru 0x01ef47 (0xe144 bytes) 000291 1788522857.623946 punch from 0x005090 to 0x00861e (0x358f bytes) 000292 1788522857.691085 write 0x015035 thru 0x01ff6b (0xaf37 bytes) 000294 1788522857.700051 trunc from 0x01ff6c to 0x005559 (0x1aa12 bytes) 000296 1788522857.734254 trunc from 0x00555a to 0x021b9c (0x1c643 bytes) 000297 1788522857.755214 read 0x00a24d thru 0x01766d (0xd421 bytes) 000298 1788522857.771755 trunc from 0x021b9d to 0x007e20 (0x19d7c bytes) 000299 1788522857.793731 punch from 0x002b88 to 0x007e20 (0x5299 bytes) 000300 1788522857.804455 MIRROR_RESYNC 4 mirrors 000301 1788522859.040152 read 0x005858 thru 0x007936 (0x20df bytes) 000302 1788522859.046130 read 0x006ed0 thru 0x007e20 (0xf51 bytes) 000303 1788522859.046458 write 0x019d85 thru 0x01c678 (0x28f4 bytes) 000306 1788522859.168177 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c679): Mirror 8: 0x8b920e2f (stale) Mirror 11: 0x8b920e2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c679): Mirror 8: 0x8b920e2f (stale) Mirror 11: 0x8b920e2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c679: skipped 000308 1788522859.438207 read 0x00c75c thru 0x013c3b (0x74e0 bytes) 000309 1788522859.453397 trunc from 0x01c679 to 0x015a7b (0x6bfd bytes) 000310 1788522859.500279 read 0x00de70 thru 0x0144ec (0x667d bytes) 000311 1788522859.513451 trunc from 0x015a7c to 0x022d78 (0xd2fd bytes) 000313 1788522859.551424 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22d79): Mirror 11: 0x99a82b58 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22d79): Mirror 11: 0x99a82b58 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22d79: skipped 000314 1788522859.883138 mapwrite 0x03658e thru 0x03ffff (0x9a72 bytes) 000315 1788522859.974800 mapwrite 0x005dcb thru 0x008d77 (0x2fad bytes) 000316 1788522860.037548 mapread 0x00d984 thru 0x018547 (0xabc4 bytes) 000322 1788522860.056702 MIRROR_RESYNC 2 mirrors 000324 1788522860.788768 write 0x02b272 thru 0x02fd36 (0x4ac5 bytes) 000325 1788522860.866862 write 0x02431d thru 0x02af7b (0x6c5f bytes) 000327 1788522860.878699 mapread 0x0196da thru 0x023cae (0xa5d5 bytes) 000330 1788522860.915379 punch from 0x02f3ed to 0x039983 (0xa597 bytes) 000331 1788522860.939131 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000332 1788522861.093154 MIRROR_EXTEND to 2 mirrors 000333 1788522862.096442 punch from 0x0046ad to 0x011b25 (0xd479 bytes) 000334 1788522862.155409 read 0x005d76 thru 0x00ce6d (0x70f8 bytes) 000335 1788522862.165277 read 0x0186c8 thru 0x01ee33 (0x676c bytes) 000336 1788522862.187066 read 0x021571 thru 0x02c0be (0xab4e bytes) 000337 1788522862.188069 mapread 0x020b4c thru 0x021e0e (0x12c3 bytes) 000339 1788522862.192585 read 0x0281ea thru 0x036989 (0xe7a0 bytes) 000340 1788522862.193789 mapwrite 0x006235 thru 0x010325 (0xa0f1 bytes) 000341 1788522862.263963 mapread 0x02f002 thru 0x033c17 (0x4c16 bytes) 000344 1788522862.269433 mapread 0x016cd6 thru 0x01b9d7 (0x4d02 bytes) 000345 1788522862.270602 trunc from 0x040000 to 0x0191f4 (0x26e0b bytes) 000352 1788522862.338155 punch from 0x002eef to 0x0104e4 (0xd5f6 bytes) 000354 1788522862.347470 punch from 0x00c51d to 0x017c8a (0xb76e bytes) 000355 1788522862.358236 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000356 1788522862.505487 mapread 0x00dd73 thru 0x0191f4 (0xb482 bytes) 000358 1788522862.554582 read 0x00e849 thru 0x0191f4 (0xa9ac bytes) 000359 1788522862.555235 trunc from 0x0191f5 to 0x0078bc (0x11938 bytes) 000361 1788522862.586181 mapread 0x006bc0 thru 0x0078bc (0xcfd bytes) 000362 1788522862.600674 read 0x00296e thru 0x0078bc (0x4f4f bytes) 000364 1788522862.602195 write 0x03e196 thru 0x03ffff (0x1e6a bytes) 000366 1788522862.607989 mapread 0x0326dc thru 0x034fae (0x28d3 bytes) 000368 1788522862.618704 mapwrite 0x02cafe thru 0x03297d (0x5e80 bytes) 000369 1788522862.703014 trunc from 0x040000 to 0x03401e (0xbfe1 bytes) 000370 1788522862.735389 trunc from 0x03401f to 0x028505 (0xbb19 bytes) 000371 1788522862.764865 mapread 0x014b93 thru 0x01c4b3 (0x7921 bytes) 000372 1788522862.770061 mapwrite 0x0287c2 thru 0x02d373 (0x4bb2 bytes) 000378 1788522862.970203 read 0x02b17d thru 0x02d373 (0x21f7 bytes) 000379 1788522862.971302 mapwrite 0x037399 thru 0x03a56f (0x31d7 bytes) 000380 1788522863.084030 punch from 0x001f5c to 0x0054af (0x3554 bytes) 000381 1788522863.091908 mapread 0x00d79d thru 0x01b902 (0xe166 bytes) 000382 1788522863.093996 punch from 0x022278 to 0x02d063 (0xadec bytes) 000385 1788522863.105618 MIRROR_EXTEND to 2 mirrors 000386 1788522863.988271 read 0x01f4f8 thru 0x02ef24 (0xfa2d bytes) 000387 1788522864.015318 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000388 1788522864.158086 mapread 0x010c1b thru 0x0158f0 (0x4cd6 bytes) 000389 1788522864.176620 mapread 0x034354 thru 0x03a56f (0x621c bytes) 000391 1788522864.186286 mapread 0x01aca7 thru 0x025a51 (0xadab bytes) 000392 1788522864.202296 read 0x005bad thru 0x009937 (0x3d8b bytes) 000397 1788522864.213145 punch from 0x01c9c8 to 0x0274e2 (0xab1b bytes) 000398 1788522864.230199 mapwrite 0x00f9a8 thru 0x01b5c5 (0xbc1e bytes) 000399 1788522864.314026 punch from 0x036308 to 0x03a56f (0x4268 bytes) 000400 1788522864.323960 punch from 0x019329 to 0x01c8d5 (0x35ad bytes) 000401 1788522864.332428 write 0x02fa2c thru 0x03a8d5 (0xaeaa bytes) 000403 1788522864.346256 mapread 0x033b4e thru 0x03a8d5 (0x6d88 bytes) 000405 1788522864.348441 punch from 0x021c30 to 0x0286e8 (0x6ab9 bytes) 000407 1788522864.363608 write 0x010ca4 thru 0x01d603 (0xc960 bytes) 000408 1788522864.365897 punch from 0x022d2b to 0x031620 (0xe8f6 bytes) 000409 1788522864.389122 trunc from 0x03a8d6 to 0x021d86 (0x18b4f bytes) 000412 1788522864.429095 punch from 0x00b58b to 0x01ab3d (0xf5b3 bytes) 000416 1788522864.438692 punch from 0x01bcf3 to 0x021d86 (0x6094 bytes) 000418 1788522864.444739 read 0x01ba01 thru 0x02057b (0x4b7b bytes) 000419 1788522864.455610 punch from 0x01f14d to 0x021d86 (0x2c3a bytes) 000420 1788522864.460249 mapread 0x00960c thru 0x014f0c (0xb901 bytes) 000421 1788522864.462765 trunc from 0x021d87 to 0x016a1c (0xb36a bytes) 000423 1788522864.484318 mapread 0x014cb4 thru 0x016a1c (0x1d69 bytes) 000425 1788522864.489166 mapwrite 0x01504c thru 0x021c6d (0xcc22 bytes) 000427 1788522864.636317 punch from 0x019e06 to 0x021c6d (0x7e68 bytes) 000428 1788522864.643248 mapread 0x016682 thru 0x021c6d (0xb5ec bytes) 000429 1788522864.693485 trunc from 0x021c6e to 0x01588c (0xc3e1 bytes) 000432 1788522864.717675 mapwrite 0x00fc6a thru 0x01e2b5 (0xe64c bytes) 000434 1788522864.843407 mapwrite 0x026f52 thru 0x02d354 (0x6403 bytes) 000437 1788522864.961496 mapwrite 0x03907d thru 0x03bcf5 (0x2c79 bytes) 000439 1788522865.081024 read 0x026fee thru 0x0311e5 (0xa1f8 bytes) 000440 1788522865.081758 read 0x0021a5 thru 0x008d7c (0x6bd8 bytes) 000441 1788522865.082879 mapwrite 0x03ec7f thru 0x03ffff (0x1381 bytes) 000442 1788522865.156095 mapwrite 0x0030eb thru 0x00deed (0xae03 bytes) 000443 1788522865.196685 punch from 0x01e5a8 to 0x021d00 (0x3759 bytes) 000444 1788522865.201497 write 0x0308ee thru 0x0329f2 (0x2105 bytes) 000446 1788522865.203697 read 0x023149 thru 0x0261b5 (0x306d bytes) 000447 1788522865.204990 write 0x00c26d thru 0x0145b7 (0x834b bytes) 000448 1788522865.207341 read 0x038a0c thru 0x03ffff (0x75f4 bytes) 000450 1788522865.208960 mapwrite 0x00d0ad thru 0x019e33 (0xcd87 bytes) 000451 1788522865.258953 punch from 0x024f1e to 0x03486e (0xf951 bytes) 000452 1788522865.265175 mapread 0x01a40b thru 0x026761 (0xc357 bytes) 000453 1788522865.277084 mapwrite 0x017733 thru 0x01baa7 (0x4375 bytes) 000456 1788522865.317337 mapwrite 0x022d5a thru 0x03202c (0xf2d3 bytes) 000458 1788522865.378959 write 0x005fca thru 0x008e00 (0x2e37 bytes) 000459 1788522865.381372 mapread 0x03c235 thru 0x03ffff (0x3dcb bytes) 000460 1788522865.385028 mapread 0x0334bc thru 0x03dc08 (0xa74d bytes) 000461 1788522865.386321 write 0x002ada thru 0x00e653 (0xbb7a bytes) 000465 1788522865.399376 mapread 0x03f96f thru 0x03ffff (0x691 bytes) 000466 1788522865.400493 punch from 0x028646 to 0x0312b1 (0x8c6c bytes) 000467 1788522865.422891 write 0x03f7f5 thru 0x03ffff (0x80b bytes) 000471 1788522865.434605 mapwrite 0x017e9b thru 0x022643 (0xa7a9 bytes) 000473 1788522865.530070 trunc from 0x040000 to 0x0368da (0x9725 bytes) 000474 1788522865.557571 read 0x033e5c thru 0x0368da (0x2a7f bytes) 000476 1788522865.561480 trunc from 0x0368db to 0x01db7f (0x18d5b bytes) 000477 1788522865.595088 mapread 0x00873b thru 0x00dd70 (0x5636 bytes) 000480 1788522865.604176 MIRROR_EXTEND to 2 mirrors 000483 1788522866.456225 trunc from 0x01db80 to 0x03d9b6 (0x1fe37 bytes) 000484 1788522866.546822 read 0x00921a thru 0x00bf15 (0x2cfc bytes) 000485 1788522866.557024 punch from 0x01786d to 0x02733a (0xface bytes) 000487 1788522866.573508 read 0x01d6d2 thru 0x0231cc (0x5afb bytes) 000489 1788522866.595589 mapwrite 0x01c677 thru 0x01cef6 (0x880 bytes) 000490 1788522866.647459 write 0x00cf07 thru 0x01cd99 (0xfe93 bytes) 000491 1788522866.656151 write 0x026e1a thru 0x03075e (0x9945 bytes) 000492 1788522866.657561 read 0x0285cf thru 0x02b2be (0x2cf0 bytes) 000493 1788522866.658198 mapwrite 0x02bf01 thru 0x037d63 (0xbe63 bytes) 000495 1788522866.756492 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d9b7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9b7): Mirror 2: 0xa31e637c (stale) CRC-32 checksum value for chunk [0, 0x3d9b7): Mirror 1: 0x71b8f6e5 Mirror 3: 0x71b8f6e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9b7: skipped 000496 1788522868.002414 mapread 0x00d375 thru 0x01144c (0x40d8 bytes) 000498 1788522868.075262 MIRROR_RESYNC 3 mirrors 000499 1788522868.852263 mapread 0x02531a thru 0x02fdb3 (0xaa9a bytes) 000500 1788522868.876051 write 0x010dfd thru 0x01d213 (0xc417 bytes) 000502 1788522869.002999 read 0x039888 thru 0x03d9b6 (0x412f bytes) 000503 1788522869.015114 punch from 0x016cbe to 0x018a6c (0x1daf bytes) 000504 1788522869.049137 trunc from 0x03d9b7 to 0x00b3bf (0x325f7 bytes) 000506 1788522869.097163 write 0x00c625 thru 0x00dda0 (0x177c bytes) 000507 1788522869.098113 punch from 0x001b31 to 0x00971a (0x7bea bytes) 000510 1788522869.111663 mapwrite 0x029923 thru 0x032bb5 (0x9293 bytes) 000512 1788522869.235400 read 0x01071d thru 0x01ad86 (0xa66a bytes) 000514 1788522869.251146 read 0x022ae8 thru 0x028d80 (0x6299 bytes) 000515 1788522869.252305 trunc from 0x032bb6 to 0x014ef5 (0x1dcc0 bytes) 000516 1788522869.274626 punch from 0x005542 to 0x012f83 (0xda42 bytes) 000517 1788522869.280725 read 0x002ec9 thru 0x00f24f (0xc387 bytes) 000518 1788522869.310148 MIRROR_RESYNC 3 mirrors 000519 1788522870.249577 trunc from 0x014ef6 to 0x01675e (0x1869 bytes) 000520 1788522870.384080 MIRROR_RESYNC 3 mirrors 000521 1788522871.643206 trunc from 0x01675f to 0x020046 (0x98e8 bytes) 000523 1788522871.727825 write 0x024942 thru 0x026029 (0x16e8 bytes) 000524 1788522871.732622 punch from 0x01eaea to 0x026029 (0x7540 bytes) 000525 1788522871.745095 mapread 0x020deb thru 0x026029 (0x523f bytes) 000526 1788522871.768623 punch from 0x01a5c5 to 0x026029 (0xba65 bytes) 000527 1788522871.780975 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2602a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2602a): Mirror 2: 0x23106176 (stale) Mirror 3: 0x23106176 (stale) CRC-32 checksum value for chunk [0, 0x2602a): Mirror 1: 0x23106176 Mirror 4: 0x23106176 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2602a: skipped 000531 1788522873.164936 trunc from 0x02602a to 0x010ca2 (0x15387 bytes) 000532 1788522873.247336 punch from 0x00011c to 0x001b20 (0x1a05 bytes) 000533 1788522873.253157 trunc from 0x010ca3 to 0x006bb9 (0xa0e9 bytes) 000534 1788522873.273105 mapread 0x002da2 thru 0x006bb9 (0x3e18 bytes) 000535 1788522873.301982 MIRROR_RESYNC 4 mirrors 000536 1788522874.264556 write 0x0165d6 thru 0x018333 (0x1d5e bytes) 000537 1788522874.336975 read 0x017cec thru 0x018111 (0x426 bytes) 000538 1788522874.356600 MIRROR_RESYNC 4 mirrors 000540 1788522875.608620 MIRROR_RESYNC 4 mirrors 000541 1788522876.118369 mapwrite 0x02100c thru 0x0255af (0x45a4 bytes) 000542 1788522876.375968 mapwrite 0x01b1ad thru 0x029710 (0xe564 bytes) 000543 1788522876.507984 mapwrite 0x000e08 thru 0x00ee16 (0xe00f bytes) 000544 1788522876.648983 punch from 0x022585 to 0x0269db (0x4457 bytes) 000545 1788522876.656200 write 0x0229d0 thru 0x022c12 (0x243 bytes) 000547 1788522876.665119 read 0x013e6d thru 0x01f61f (0xb7b3 bytes) 000549 1788522876.678003 mapwrite 0x01fb54 thru 0x0286f4 (0x8ba1 bytes) 000550 1788522876.751685 punch from 0x00cff3 to 0x00dfd9 (0xfe7 bytes) 000551 1788522876.763945 trunc from 0x029711 to 0x02d7fd (0x40ed bytes) 000552 1788522876.787447 trunc from 0x02d7fe to 0x021824 (0xbfd9 bytes) 000553 1788522876.808021 read 0x010224 thru 0x010fb6 (0xd93 bytes) 000554 1788522876.818936 mapwrite 0x00ccfe thru 0x012078 (0x537b bytes) 000555 1788522876.871737 trunc from 0x021825 to 0x03d12f (0x1b90b bytes) 000556 1788522876.895397 write 0x00cd97 thru 0x01023d (0x34a7 bytes) 000557 1788522876.905807 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d130): Mirror 2: 0xb6b654cf (stale) Mirror 3: 0xb6b654cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d130): Mirror 2: 0xb6b654cf (stale) Mirror 3: 0xb6b654cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d130: skipped 000558 1788522877.314266 MIRROR_RESYNC 3 mirrors 000561 1788522878.322832 mapread 0x007e0a thru 0x00f573 (0x776a bytes) 000562 1788522878.349574 write 0x0304cb thru 0x030e3c (0x972 bytes) 000563 1788522878.440805 write 0x02cf87 thru 0x02d0ba (0x134 bytes) 000564 1788522878.452836 write 0x00158d thru 0x0103a5 (0xee19 bytes) 000565 1788522878.469315 mapwrite 0x035160 thru 0x03ffff (0xaea0 bytes) 000566 1788522878.605942 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3a54ea9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x3a54ea9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000567 1788522878.924366 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000568 1788522879.057128 trunc from 0x040000 to 0x039327 (0x6cd8 bytes) 000571 1788522879.077519 mapwrite 0x027c5f thru 0x0318de (0x9c80 bytes) 000572 1788522879.245746 mapwrite 0x02494d thru 0x0326ab (0xdd5f bytes) 000573 1788522879.371981 mapwrite 0x01c6a8 thru 0x027e8e (0xb7e7 bytes) 000574 1788522879.527631 punch from 0x0215b3 to 0x02cd03 (0xb751 bytes) 000575 1788522879.542386 punch from 0x0371fc to 0x039327 (0x212c bytes) 000576 1788522879.556338 punch from 0x00ceb0 to 0x011aa9 (0x4bfa bytes) 000577 1788522879.579338 mapwrite 0x01afae thru 0x026af0 (0xbb43 bytes) 000580 1788522879.723061 write 0x03f4b5 thru 0x03ffff (0xb4b bytes) 000583 1788522879.734116 mapwrite 0x009228 thru 0x012090 (0x8e69 bytes) 000584 1788522879.880045 mapwrite 0x00d695 thru 0x017764 (0xa0d0 bytes) 000585 1788522879.950299 mapwrite 0x0292f5 thru 0x02f17f (0x5e8b bytes) 000586 1788522880.054961 read 0x030871 thru 0x0394d5 (0x8c65 bytes) 000587 1788522880.071997 write 0x00f135 thru 0x01ea25 (0xf8f1 bytes) 000588 1788522880.080747 mapread 0x033d8c thru 0x0349ad (0xc22 bytes) 000589 1788522880.084803 trunc from 0x040000 to 0x02bf51 (0x140ae bytes) 000590 1788522880.140127 punch from 0x013890 to 0x014b6d (0x12de bytes) 000591 1788522880.165404 write 0x007d56 thru 0x010b00 (0x8dab bytes) 000592 1788522880.166712 mapread 0x00ec47 thru 0x0132ad (0x4667 bytes) 000596 1788522880.186841 mapread 0x00b366 thru 0x01068c (0x5327 bytes) 000597 1788522880.188277 mapwrite 0x02b33d thru 0x02db65 (0x2829 bytes) 000598 1788522880.376740 read 0x01d2e1 thru 0x029cc3 (0xc9e3 bytes) 000600 1788522880.377958 trunc from 0x02db66 to 0x008b61 (0x25004 bytes) 000601 1788522880.445507 mapread 0x00087c thru 0x0036bc (0x2e41 bytes) 000604 1788522880.466599 write 0x026e46 thru 0x02bee1 (0x509c bytes) 000605 1788522880.470384 mapread 0x0244c0 thru 0x02bee1 (0x7a22 bytes) 000606 1788522880.494233 write 0x00fcf6 thru 0x01899f (0x8caa bytes) 000609 1788522880.500208 trunc from 0x02bee2 to 0x01c4e2 (0xf9ff bytes) 000610 1788522880.553798 read 0x016cbc thru 0x019dcc (0x3111 bytes) 000612 1788522880.560445 mapwrite 0x01efaa thru 0x02df35 (0xef8c bytes) 000613 1788522880.789398 write 0x007e89 thru 0x0105a8 (0x8720 bytes) 000614 1788522880.790891 trunc from 0x02df36 to 0x02f9f7 (0x1ac2 bytes) 000618 1788522880.839262 mapwrite 0x03affc thru 0x03ffff (0x5004 bytes) 000619 1788522880.936258 mapwrite 0x023dd4 thru 0x02d4ad (0x96da bytes) 000620 1788522880.986246 mapwrite 0x00aeb3 thru 0x00dc3b (0x2d89 bytes) 000621 1788522881.027554 read 0x010d72 thru 0x012d7f (0x200e bytes) 000624 1788522881.036540 trunc from 0x040000 to 0x014e7e (0x2b181 bytes) 000625 1788522881.081017 trunc from 0x014e7f to 0x014a6a (0x414 bytes) 000626 1788522881.121441 write 0x02c84c thru 0x0302e4 (0x3a99 bytes) 000627 1788522881.123437 mapwrite 0x01b8a8 thru 0x01c5af (0xd08 bytes) 000628 1788522881.195935 read 0x024e75 thru 0x02a429 (0x55b5 bytes) 000630 1788522881.198853 trunc from 0x0302e5 to 0x00031a (0x2ffca bytes) 000631 1788522881.232108 trunc from 0x00031b to 0x01f889 (0x1f56f bytes) 000633 1788522881.254051 write 0x02f811 thru 0x03663b (0x6e2b bytes) 000634 1788522881.256267 punch from 0x02c82a to 0x032df1 (0x65c8 bytes) 000637 1788522881.276275 read 0x027788 thru 0x032afe (0xb377 bytes) 000638 1788522881.331179 punch from 0x0086e6 to 0x009a6e (0x1389 bytes) 000639 1788522881.340068 trunc from 0x03663c to 0x003979 (0x32cc2 bytes) 000640 1788522881.363807 mapread 0x000967 thru 0x003979 (0x3013 bytes) 000643 1788522881.377076 MIRROR_EXTEND to 2 mirrors 000644 1788522882.429075 write 0x02c11c thru 0x02d285 (0x116a bytes) 000645 1788522882.515465 MIRROR_RESYNC 2 mirrors 000648 1788522883.731951 read 0x02a4f7 thru 0x02ca0e (0x2518 bytes) 000649 1788522883.738310 mapread 0x009926 thru 0x015c08 (0xc2e3 bytes) 000651 1788522883.766487 mapread 0x004dea thru 0x00ddde (0x8ff5 bytes) 000652 1788522883.767368 mapwrite 0x0326d4 thru 0x03ffff (0xd92c bytes) 000654 1788522884.027612 trunc from 0x040000 to 0x00ee5f (0x311a0 bytes) 000656 1788522884.055454 write 0x03ebea thru 0x03ffff (0x1416 bytes) 000657 1788522884.060183 mapread 0x015381 thru 0x01cc4d (0x78cd bytes) 000658 1788522884.090196 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000660 1788522884.301228 mapwrite 0x014976 thru 0x01ffaa (0xb635 bytes) 000661 1788522884.394361 punch from 0x0120bd to 0x013699 (0x15dd bytes) 000664 1788522884.405872 punch from 0x006748 to 0x00f7ec (0x90a5 bytes) 000667 1788522884.420859 mapwrite 0x038834 thru 0x03ea50 (0x621d bytes) 000669 1788522884.502839 read 0x028b01 thru 0x029895 (0xd95 bytes) 000670 1788522884.509083 mapwrite 0x025df1 thru 0x02b97b (0x5b8b bytes) 000671 1788522884.570744 punch from 0x0037bb to 0x004a42 (0x1288 bytes) 000672 1788522884.576887 mapwrite 0x03ca27 thru 0x03ffff (0x35d9 bytes) 000675 1788522884.672152 punch from 0x02d3c8 to 0x033a3f (0x6678 bytes) 000679 1788522884.681557 read 0x015aab thru 0x018bee (0x3144 bytes) 000680 1788522884.683060 mapread 0x007a20 thru 0x014992 (0xcf73 bytes) 000681 1788522884.696220 mapread 0x036a77 thru 0x03dfed (0x7577 bytes) 000683 1788522884.703906 MIRROR_EXTEND to 2 mirrors 000684 1788522885.852024 mapread 0x03f58a thru 0x03ffff (0xa76 bytes) 000685 1788522885.880112 MIRROR_EXTEND to 3 mirrors 000686 1788522886.716789 mapwrite 0x01cd73 thru 0x024dbf (0x804d bytes) 000687 1788522886.944898 mapread 0x039c13 thru 0x03ffff (0x63ed bytes) 000688 1788522886.973969 read 0x032c84 thru 0x039a9c (0x6e19 bytes) 000691 1788522886.977612 punch from 0x03ada2 to 0x03ffff (0x525e bytes) 000692 1788522886.994728 read 0x000ca7 thru 0x0069bd (0x5d17 bytes) 000693 1788522887.012099 trunc from 0x040000 to 0x030227 (0xfdd8 bytes) 000694 1788522887.060000 mapread 0x0252f8 thru 0x025579 (0x282 bytes) 000696 1788522887.066985 trunc from 0x030228 to 0x01033a (0x1feed bytes) 000698 1788522887.121221 punch from 0x00b8f3 to 0x01033a (0x4a48 bytes) 000700 1788522887.128079 read 0x002f73 thru 0x00cabe (0x9b4c bytes) 000702 1788522887.138501 mapread 0x00bc12 thru 0x01033a (0x4729 bytes) 000703 1788522887.139686 mapwrite 0x030e4d thru 0x0341c4 (0x3378 bytes) 000704 1788522887.217533 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x341c5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x341c5): Mirror 2: 0xaab15ac4 (stale) Mirror 3: 0xaab15ac4 (stale) CRC-32 checksum value for chunk [0, 0x341c5): Mirror 1: 0x4959107e Mirror 4: 0x4959107e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x341c5: skipped 000705 1788522888.561213 trunc from 0x0341c5 to 0x00ad2d (0x29497 bytes) 000707 1788522888.673095 mapwrite 0x011f0d thru 0x01b21e (0x9312 bytes) 000709 1788522888.820697 mapwrite 0x017429 thru 0x01f990 (0x8568 bytes) 000710 1788522888.904586 mapwrite 0x0150dc thru 0x01c54e (0x7473 bytes) 000711 1788522888.956464 mapread 0x019698 thru 0x01d05f (0x39c8 bytes) 000713 1788522888.962264 punch from 0x0088b6 to 0x009d14 (0x145f bytes) 000714 1788522888.968126 trunc from 0x01f991 to 0x02a718 (0xad88 bytes) 000715 1788522888.993049 punch from 0x00d2d1 to 0x01951f (0xc24f bytes) 000716 1788522889.004109 mapread 0x008c71 thru 0x01414d (0xb4dd bytes) 000717 1788522889.032162 mapwrite 0x032ab7 thru 0x03afab (0x84f5 bytes) 000718 1788522889.132381 punch from 0x02871c to 0x02b213 (0x2af8 bytes) 000720 1788522889.170183 read 0x028c52 thru 0x02ec32 (0x5fe1 bytes) 000721 1788522889.200611 MIRROR_RESYNC 4 mirrors 000722 1788522890.841422 write 0x02742a thru 0x02ba27 (0x45fe bytes) 000723 1788522890.915222 MIRROR_RESYNC 4 mirrors 000724 1788522892.528962 punch from 0x00e51e to 0x0184db (0x9fbe bytes) 000726 1788522892.613459 mapwrite 0x02775e thru 0x02fb61 (0x8404 bytes) 000727 1788522892.723573 MIRROR_RESYNC 4 mirrors 000728 1788522894.162167 read 0x012556 thru 0x01c411 (0x9ebc bytes) 000729 1788522894.172494 mapwrite 0x03f897 thru 0x03ffff (0x769 bytes) 000730 1788522894.344847 mapread 0x00c452 thru 0x01ad20 (0xe8cf bytes) 000732 1788522894.380456 read 0x03a68d thru 0x03ffff (0x5973 bytes) 000733 1788522894.403805 read 0x03e04a thru 0x03ffff (0x1fb6 bytes) 000734 1788522894.404536 mapwrite 0x02342e thru 0x02aa69 (0x763c bytes) 000735 1788522894.485717 mapread 0x0241f9 thru 0x026b72 (0x297a bytes) 000737 1788522894.488973 trunc from 0x040000 to 0x02f8a9 (0x10756 bytes) 000738 1788522894.555020 trunc from 0x02f8aa to 0x034bc6 (0x531d bytes) 000739 1788522894.599528 trunc from 0x034bc7 to 0x00876f (0x2c457 bytes) 000740 1788522894.642394 mapwrite 0x0324b2 thru 0x03f611 (0xd160 bytes) 000741 1788522894.782613 read 0x024a9f thru 0x031b0c (0xd06e bytes) 000742 1788522894.783293 read 0x0252a1 thru 0x02c0d2 (0x6e32 bytes) 000743 1788522894.784522 trunc from 0x03f612 to 0x039d7e (0x5893 bytes) 000745 1788522894.812536 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39d7f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d7f): Mirror 2: 0xa1c695d0 (stale) Mirror 3: 0xa1c695d0 (stale) Mirror 4: 0xa1c695d0 (stale) CRC-32 checksum value for chunk [0, 0x39d7f): Mirror 1: 0x2da0bc7c Mirror 5: 0x2da0bc7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d7f: skipped 000747 1788522896.195984 punch from 0x00653f to 0x00b93d (0x53ff bytes) 000748 1788522896.260045 mapread 0x0066a9 thru 0x00e191 (0x7ae9 bytes) 000750 1788522896.276922 punch from 0x0300d3 to 0x0389e5 (0x8913 bytes) 000751 1788522896.281467 punch from 0x0155fd to 0x01ec0f (0x9613 bytes) 000752 1788522896.287125 mapwrite 0x00c515 thru 0x011c67 (0x5753 bytes) 000753 1788522896.362903 mapread 0x028b30 thru 0x02a1d0 (0x16a1 bytes) 000756 1788522896.375273 punch from 0x005c8e to 0x014ccf (0xf042 bytes) 000757 1788522896.380878 mapwrite 0x0246fb thru 0x02e555 (0x9e5b bytes) 000758 1788522896.440904 write 0x00e44c thru 0x019953 (0xb508 bytes) 000759 1788522896.447133 trunc from 0x039d7f to 0x0293fa (0x10984 bytes) 000761 1788522896.470025 write 0x000cad thru 0x00b43b (0xa78f bytes) 000762 1788522896.492269 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x293fb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x293fb): Mirror 2: 0x39882b6a (stale) Mirror 3: 0x39882b6a (stale) Mirror 4: 0x39882b6a (stale) Mirror 5: 0x97e83bed (stale) CRC-32 checksum value for chunk [0, 0x293fb): Mirror 1: 0x9d393392 Mirror 6: 0x9d393392 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x293fb: skipped 000763 1788522897.760968 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x293fb): Mirror 2: 0x39882b6a (stale) Mirror 3: 0x39882b6a (stale) Mirror 5: 0x97e83bed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x293fb): Mirror 2: 0x39882b6a (stale) Mirror 3: 0x39882b6a (stale) Mirror 5: 0x97e83bed (stale) CRC-32 checksum value for chunk [0, 0x293fb): Mirror 1: 0x9d393392 Mirror 6: 0x9d393392 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x293fb: skipped 000764 1788522898.455819 mapread 0x00dabc thru 0x01b400 (0xd945 bytes) 000767 1788522898.499821 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x293fb): Mirror 2: 0x39882b6a (stale) Mirror 5: 0x97e83bed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x293fb): Mirror 2: 0x39882b6a (stale) Mirror 5: 0x97e83bed (stale) CRC-32 checksum value for chunk [0, 0x293fb): Mirror 1: 0x9d393392 Mirror 6: 0x9d393392 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x293fb: skipped 000768 1788522899.609782 MIRROR_RESYNC 4 mirrors 000769 1788522900.662009 trunc from 0x0293fb to 0x03048d (0x7093 bytes) 000770 1788522900.754562 trunc from 0x03048e to 0x01a0de (0x163af bytes) 000771 1788522900.780430 read 0x00a429 thru 0x010fdf (0x6bb7 bytes) 000773 1788522900.791444 trunc from 0x01a0df to 0x029574 (0xf496 bytes) 000776 1788522900.819805 write 0x02f2df thru 0x034f8a (0x5cac bytes) 000777 1788522900.830438 MIRROR_RESYNC 4 mirrors 000778 1788522902.166530 trunc from 0x034f8b to 0x027773 (0xd817 bytes) 000779 1788522902.236421 write 0x002e78 thru 0x008536 (0x56bf bytes) 000780 1788522902.248458 write 0x028422 thru 0x03078c (0x836b bytes) 000781 1788522902.249895 read 0x0132d7 thru 0x017016 (0x3d40 bytes) 000782 1788522902.265878 MIRROR_RESYNC 4 mirrors 000783 1788522904.037378 MIRROR_EXTEND to 5 mirrors 000784 1788522904.926717 MIRROR_RESYNC 5 mirrors 000788 1788522905.405845 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) 000789 1788522905.789841 mapread 0x0166d1 thru 0x01eaf0 (0x8420 bytes) 000790 1788522905.824603 read 0x02d005 thru 0x03078c (0x3788 bytes) 000791 1788522905.831861 mapread 0x008862 thru 0x0102f0 (0x7a8f bytes) 000792 1788522905.843974 trunc from 0x03078d to 0x00502a (0x2b762 bytes) 000793 1788522905.928349 mapwrite 0x001922 thru 0x002969 (0x1048 bytes) 000796 1788522905.991236 write 0x00dfb4 thru 0x017b59 (0x9ba6 bytes) 000798 1788522905.994482 mapread 0x009202 thru 0x016d6f (0xdb6e bytes) 000799 1788522906.002424 mapread 0x016c3d thru 0x017b59 (0xf1d bytes) 000800 1788522906.012485 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17b5a): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17b5a): Mirror 2: 0x2eea3ea9 (stale) Mirror 5: 0x2eea3ea9 (stale) Mirror 6: 0x2eea3ea9 (stale) CRC-32 checksum value for chunk [0, 0x17b5a): Mirror 1: 0xd3a3575b Mirror 7: 0xd3a3575b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17b5a: skipped 000801 1788522907.123959 write 0x012cfd thru 0x01342f (0x733 bytes) 000802 1788522907.231263 write 0x036892 thru 0x03d8e4 (0x7053 bytes) 000805 1788522907.238311 mapwrite 0x01aeab thru 0x02232f (0x7485 bytes) 000806 1788522907.351075 mapwrite 0x016bc9 thru 0x018125 (0x155d bytes) 000808 1788522907.456924 punch from 0x0234be to 0x029afe (0x6641 bytes) 000809 1788522907.472436 mapwrite 0x02523d thru 0x0327c1 (0xd585 bytes) 000811 1788522907.586911 trunc from 0x03d8e5 to 0x0104e0 (0x2d404 bytes) 000813 1788522907.620037 read 0x007f12 thru 0x0104e0 (0x85cf bytes) 000814 1788522907.633704 punch from 0x006913 to 0x0104e0 (0x9bce bytes) 000816 1788522907.646787 MIRROR_RESYNC 5 mirrors 000818 1788522909.059360 read 0x00dfe4 thru 0x0104e0 (0x24fd bytes) 000820 1788522909.070815 MIRROR_EXTEND to 6 mirrors 000821 1788522910.309870 trunc from 0x0104e1 to 0x0350d3 (0x24bf3 bytes) 000822 1788522910.418883 mapread 0x0145ee thru 0x01643f (0x1e52 bytes) 000823 1788522910.443623 punch from 0x00fbc0 to 0x01eb70 (0xefb1 bytes) 000826 1788522910.451780 write 0x0128d9 thru 0x020429 (0xdb51 bytes) 000827 1788522910.483286 mapread 0x019773 thru 0x01ad8f (0x161d bytes) 000828 1788522910.498807 MIRROR_RESYNC 6 mirrors 000830 1788522912.747426 read 0x012b5a thru 0x014d07 (0x21ae bytes) 000832 1788522912.753382 trunc from 0x0350d4 to 0x0125d5 (0x22afe bytes) 000833 1788522912.888072 punch from 0x006659 to 0x0125d5 (0xbf7d bytes) 000834 1788522912.910716 trunc from 0x0125d6 to 0x0291a2 (0x16bcd bytes) 000835 1788522912.937935 mapwrite 0x017053 thru 0x025b87 (0xeb35 bytes) 000836 1788522913.108292 write 0x01137a thru 0x016655 (0x52dc bytes) 000837 1788522913.113802 read 0x0217cf thru 0x0279cf (0x6201 bytes) 000838 1788522913.118684 trunc from 0x0291a3 to 0x00dccc (0x1b4d6 bytes) 000839 1788522913.147945 trunc from 0x00dccd to 0x00a272 (0x3a5a bytes) 000840 1788522913.172698 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xa273): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa273): Mirror 2: 0x320e2548 (stale) Mirror 5: 0x320e2548 (stale) Mirror 6: 0x320e2548 (stale) Mirror 7: 0x320e2548 (stale) Mirror 8: 0x320e2548 (stale) CRC-32 checksum value for chunk [0, 0xa273): Mirror 1: 0x320e2548 Mirror 9: 0x320e2548 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa273: skipped 000841 1788522914.776901 mapwrite 0x0216da thru 0x026e1b (0x5742 bytes) 000844 1788522915.060944 read 0x001489 thru 0x004755 (0x32cd bytes) 000845 1788522915.073267 read 0x019f1f thru 0x01a9de (0xac0 bytes) 000846 1788522915.101769 write 0x037ea4 thru 0x03c391 (0x44ee bytes) 000848 1788522915.103408 punch from 0x019825 to 0x01bbbc (0x2398 bytes) 000849 1788522915.141300 punch from 0x036c83 to 0x03b0ee (0x446c bytes) 000850 1788522915.152523 punch from 0x027961 to 0x02e884 (0x6f24 bytes) 000855 1788522915.172884 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c392): Mirror 2: 0x76c3d8b8 (stale) Mirror 5: 0x76c3d8b8 (stale) Mirror 6: 0x76c3d8b8 (stale) Mirror 7: 0x76c3d8b8 (stale) Mirror 9: 0xb3f2cffa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c392): Mirror 2: 0x76c3d8b8 (stale) Mirror 5: 0x76c3d8b8 (stale) Mirror 6: 0x76c3d8b8 (stale) Mirror 7: 0x76c3d8b8 (stale) Mirror 9: 0xb3f2cffa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c392: skipped 000856 1788522915.670947 trunc from 0x03c392 to 0x0311f1 (0xb1a0 bytes) 000857 1788522915.703550 mapread 0x01f0be thru 0x02a8ef (0xb832 bytes) 000858 1788522915.740484 punch from 0x030a59 to 0x0311f1 (0x799 bytes) 000862 1788522915.747475 write 0x034029 thru 0x03ffff (0xbfd7 bytes) 000863 1788522915.749771 read 0x00d0cb thru 0x01446a (0x73a0 bytes) 000864 1788522915.756854 punch from 0x00b1f5 to 0x00d11d (0x1f29 bytes) 000865 1788522915.767842 mapread 0x008499 thru 0x00c461 (0x3fc9 bytes) 000867 1788522915.777773 mapread 0x00d73d thru 0x014903 (0x71c7 bytes) 000870 1788522915.779793 trunc from 0x040000 to 0x01a4ba (0x25b45 bytes) 000871 1788522915.793592 punch from 0x000cdb to 0x00d54b (0xc871 bytes) 000872 1788522915.803759 read 0x00079f thru 0x00636a (0x5bcc bytes) 000873 1788522915.828309 mapwrite 0x01679f thru 0x01f011 (0x8873 bytes) 000875 1788522915.928446 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1f012): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f012): Mirror 2: 0x15162523 (stale) Mirror 5: 0x15162523 (stale) Mirror 6: 0x15162523 (stale) Mirror 7: 0x15162523 (stale) Mirror 9: 0x8e4442df (stale) CRC-32 checksum value for chunk [0, 0x1f012): Mirror 1: 0x7cd000f1 Mirror 10: 0x7cd000f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f012: skipped 000876 1788522917.440277 mapwrite 0x017715 thru 0x0250db (0xd9c7 bytes) 000877 1788522917.641488 mapwrite 0x036476 thru 0x03b6b3 (0x523e bytes) 000878 1788522917.727044 punch from 0x00e1ca to 0x01b3f4 (0xd22b bytes) 000879 1788522917.734736 read 0x01028b thru 0x017716 (0x748c bytes) 000880 1788522917.767560 punch from 0x024bc9 to 0x02d0d0 (0x8508 bytes) 000881 1788522917.773799 mapread 0x0227b8 thru 0x02d0fd (0xa946 bytes) 000882 1788522917.796490 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b6b4): Mirror 2: 0x480e3472 (stale) Mirror 5: 0x480e3472 (stale) Mirror 6: 0x480e3472 (stale) Mirror 9: 0xdbd3bc5 (stale) Mirror 10: 0x4488fb28 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6b4): Mirror 2: 0x480e3472 (stale) Mirror 5: 0x480e3472 (stale) Mirror 6: 0x480e3472 (stale) Mirror 9: 0xdbd3bc5 (stale) Mirror 10: 0x4488fb28 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6b4: skipped 000883 1788522918.251081 read 0x02fdb0 thru 0x03b6b3 (0xb904 bytes) 000885 1788522918.272913 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b6b4): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6b4): Mirror 2: 0x480e3472 (stale) Mirror 5: 0x480e3472 (stale) Mirror 6: 0x480e3472 (stale) Mirror 9: 0xdbd3bc5 (stale) Mirror 10: 0x4488fb28 (stale) CRC-32 checksum value for chunk [0, 0x3b6b4): Mirror 1: 0xda1b4060 Mirror 11: 0xda1b4060 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6b4: skipped 000887 1788522919.487577 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b6b4): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b6b4): Mirror 2: 0x480e3472 (stale) Mirror 5: 0x480e3472 (stale) Mirror 6: 0x480e3472 (stale) Mirror 9: 0xdbd3bc5 (stale) Mirror 10: 0x4488fb28 (stale) CRC-32 checksum value for chunk [0, 0x3b6b4): Mirror 1: 0xda1b4060 Mirror 11: 0xda1b4060 Mirror 12: 0xda1b4060 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b6b4: skipped 000889 1788522920.316219 trunc from 0x03b6b4 to 0x030cfb (0xa9b8 bytes) 000892 1788522920.419200 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30cfc): Mirror 5: 0xf49ab5a9 (stale) Mirror 6: 0xf49ab5a9 (stale) Mirror 9: 0xbc9a86c3 (stale) Mirror 10: 0x70108817 (stale) Mirror 11: 0xd23ef752 (stale) Mirror 12: 0xd23ef752 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cfc): Mirror 5: 0xf49ab5a9 (stale) Mirror 6: 0xf49ab5a9 (stale) Mirror 9: 0xbc9a86c3 (stale) Mirror 10: 0x70108817 (stale) Mirror 11: 0xd23ef752 (stale) Mirror 12: 0xd23ef752 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cfc: skipped 000893 1788522920.886426 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30cfc): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30cfc): Mirror 5: 0xf49ab5a9 (stale) Mirror 6: 0xf49ab5a9 (stale) Mirror 9: 0xbc9a86c3 (stale) Mirror 10: 0x70108817 (stale) Mirror 11: 0xd23ef752 (stale) Mirror 12: 0xd23ef752 (stale) CRC-32 checksum value for chunk [0, 0x30cfc): Mirror 1: 0xd23ef752 Mirror 13: 0xd23ef752 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30cfc: skipped 000894 1788522922.178987 punch from 0x01b77b to 0x01bec9 (0x74f bytes) 000897 1788522922.333766 mapwrite 0x015b01 thru 0x022cd4 (0xd1d4 bytes) 000898 1788522922.503116 mapwrite 0x01a74f thru 0x01bf70 (0x1822 bytes) 000899 1788522922.614433 write 0x007e65 thru 0x00d5e7 (0x5783 bytes) 000900 1788522922.635815 read 0x0085ed thru 0x00beda (0x38ee bytes) 000901 1788522922.636766 mapread 0x00e5f3 thru 0x015d02 (0x7710 bytes) 000904 1788522922.660003 mapwrite 0x037976 thru 0x03ffff (0x868a bytes) 000907 1788522922.811326 mapread 0x015053 thru 0x01744c (0x23fa bytes) 000908 1788522922.812257 punch from 0x00dcbd to 0x0186ca (0xaa0e bytes) 000910 1788522922.840366 punch from 0x0082cc to 0x00cfc4 (0x4cf9 bytes) 000912 1788522922.856017 mapwrite 0x03b0c0 thru 0x03ffff (0x4f40 bytes) 000915 1788522922.956033 mapread 0x006231 thru 0x00c830 (0x6600 bytes) 000916 1788522922.988265 read 0x02f7f8 thru 0x03d656 (0xde5f bytes) 000918 1788522923.017825 write 0x00c084 thru 0x0185c6 (0xc543 bytes) 000920 1788522923.022306 mapwrite 0x01578a thru 0x01c60e (0x6e85 bytes) 000921 1788522923.147187 trunc from 0x040000 to 0x028d51 (0x172ae bytes) 000922 1788522923.193182 mapwrite 0x024394 thru 0x02a37b (0x5fe8 bytes) 000924 1788522923.283415 write 0x03de07 thru 0x03ffff (0x21f9 bytes) 000925 1788522923.286432 punch from 0x0083ee to 0x0093a6 (0xfb9 bytes) 000926 1788522923.299998 write 0x00a92a thru 0x00eb79 (0x4250 bytes) 000927 1788522923.302323 write 0x00185e thru 0x0020b7 (0x85a bytes) 000928 1788522923.303631 mapread 0x004672 thru 0x014667 (0xfff6 bytes) 000929 1788522923.319898 trunc from 0x040000 to 0x000be2 (0x3f41d bytes) 000930 1788522923.346070 punch from 0x0008e5 to 0x000be2 (0x2fe bytes) 000933 1788522923.355013 trunc from 0x000be3 to 0x0397fe (0x38c1c bytes) 000934 1788522923.378847 mapread 0x025ccb thru 0x02714a (0x1480 bytes) 000936 1788522923.414500 mapwrite 0x013c24 thru 0x014fdc (0x13b9 bytes) 000937 1788522923.520806 MIRROR_RESYNC 8 mirrors 000938 1788522925.937794 mapread 0x007d15 thru 0x01374a (0xba36 bytes) 000941 1788522925.996353 mapread 0x030eed thru 0x033e45 (0x2f59 bytes) 000943 1788522926.030893 mapread 0x01b168 thru 0x01ea9d (0x3936 bytes) 000944 1788522926.060273 trunc from 0x0397ff to 0x026485 (0x13379 bytes) 000945 1788522926.281367 trunc from 0x026486 to 0x0327f3 (0xc36e bytes) 000946 1788522926.311239 trunc from 0x0327f4 to 0x007ab1 (0x2ad42 bytes) 000947 1788522926.340019 punch from 0x002f54 to 0x007ab1 (0x4b5e bytes) 000948 1788522926.354032 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7ab2): Mirror 5: 0x44538cfe (stale) Mirror 6: 0x44538cfe (stale) Mirror 9: 0x44538cfe (stale) Mirror 10: 0x44538cfe (stale) Mirror 11: 0x44538cfe (stale) Mirror 13: 0x44538cfe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7ab2): Mirror 5: 0x44538cfe (stale) Mirror 6: 0x44538cfe (stale) Mirror 9: 0x44538cfe (stale) Mirror 10: 0x44538cfe (stale) Mirror 11: 0x44538cfe (stale) Mirror 13: 0x44538cfe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7ab2: skipped 000949 1788522926.875887 mapread 0x002c76 thru 0x00444b (0x17d6 bytes) 000950 1788522926.898756 punch from 0x0018d6 to 0x003e18 (0x2543 bytes) 000952 1788522926.903915 mapread 0x006267 thru 0x007ab1 (0x184b bytes) 000953 1788522926.905276 write 0x025ff0 thru 0x0298ea (0x38fb bytes) 000955 1788522926.908991 write 0x02f04b thru 0x03da14 (0xe9ca bytes) 000956 1788522926.911418 write 0x02e889 thru 0x02ed2c (0x4a4 bytes) 000957 1788522926.923201 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3da15): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da15): Mirror 5: 0x3dbbbdd0 (stale) Mirror 6: 0x3dbbbdd0 (stale) Mirror 9: 0x3dbbbdd0 (stale) Mirror 10: 0x3dbbbdd0 (stale) Mirror 11: 0x3dbbbdd0 (stale) Mirror 13: 0x3dbbbdd0 (stale) CRC-32 checksum value for chunk [0, 0x3da15): Mirror 1: 0x6da3ba32 Mirror 14: 0x6da3ba32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da15: skipped 000958 1788522928.346046 read 0x00e1e0 thru 0x01d548 (0xf369 bytes) 000961 1788522928.450683 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3da15): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da15): Mirror 5: 0x3dbbbdd0 (stale) Mirror 6: 0x3dbbbdd0 (stale) Mirror 9: 0x3dbbbdd0 (stale) Mirror 10: 0x3dbbbdd0 (stale) Mirror 11: 0x3dbbbdd0 (stale) Mirror 13: 0x3dbbbdd0 (stale) CRC-32 checksum value for chunk [0, 0x3da15): Mirror 1: 0x6da3ba32 Mirror 14: 0x6da3ba32 Mirror 15: 0x6da3ba32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da15: skipped 000962 1788522929.555759 trunc from 0x03da15 to 0x02c6b9 (0x1135b bytes) 000964 1788522929.680347 write 0x019963 thru 0x023d31 (0xa3cf bytes) 000965 1788522929.696705 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2c6ba): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c6ba): Mirror 5: 0xf8c2243 (stale) Mirror 6: 0xf8c2243 (stale) Mirror 9: 0xf8c2243 (stale) Mirror 10: 0xf8c2243 (stale) Mirror 11: 0xf8c2243 (stale) Mirror 13: 0xf8c2243 (stale) Mirror 14: 0x8e550ea4 (stale) Mirror 15: 0x8e550ea4 (stale) CRC-32 checksum value for chunk [0, 0x2c6ba): Mirror 1: 0x62d0812b Mirror 16: 0x62d0812b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c6ba: skipped 000966 1788522930.740363 write 0x018054 thru 0x01d737 (0x56e4 bytes) 000967 1788522930.852596 punch from 0x01890a to 0x0232ac (0xa9a3 bytes) 000969 1788522930.879752 mapwrite 0x01bfe2 thru 0x028bc8 (0xcbe7 bytes) 000970 1788522930.987481 mapread 0x007efa thru 0x0119fc (0x9b03 bytes) 000971 1788522931.026578 trunc from 0x02c6ba to 0x037720 (0xb067 bytes) 000973 1788522931.065685 mapwrite 0x0165bf thru 0x020153 (0x9b95 bytes) 000975 1788522931.158263 read 0x01ff9d thru 0x027f03 (0x7f67 bytes) 000976 1788522931.158871 trunc from 0x037721 to 0x00fbbf (0x27b61 bytes) 000977 1788522931.186232 mapread 0x0029fa thru 0x00ad36 (0x833d bytes) 000978 1788522931.187360 write 0x03a092 thru 0x03ffff (0x5f6e bytes) 000980 1788522931.191634 read 0x032ec9 thru 0x03dede (0xb016 bytes) 000981 1788522931.210975 mapread 0x009dc4 thru 0x018553 (0xe790 bytes) 000982 1788522931.213258 read 0x032ce3 thru 0x03c7e6 (0x9b04 bytes) 000986 1788522931.218086 mapwrite 0x00acb2 thru 0x01a7fe (0xfb4d bytes) 000987 1788522931.322469 write 0x03a549 thru 0x03ba52 (0x150a bytes) 000989 1788522931.323898 read 0x02bde5 thru 0x02f44b (0x3667 bytes) 000990 1788522931.324455 punch from 0x002338 to 0x00feae (0xdb77 bytes) 000991 1788522931.350870 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xa5dca638 (stale) Mirror 6: 0xa5dca638 (stale) Mirror 9: 0xa5dca638 (stale) Mirror 10: 0xa5dca638 (stale) Mirror 11: 0xa5dca638 (stale) Mirror 13: 0xa5dca638 (stale) Mirror 14: 0xb35f511 (stale) Mirror 15: 0xb35f511 (stale) Mirror 16: 0x7c01a678 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe531c82c Mirror 17: 0xe531c82c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000992 1788522932.629189 trunc from 0x040000 to 0x0039a3 (0x3c65c bytes) 000993 1788522932.857830 mapread 0x002f79 thru 0x003500 (0x588 bytes) 000996 1788522932.885794 write 0x00341c thru 0x009392 (0x5f77 bytes) 000997 1788522932.890360 mapread 0x0046a4 thru 0x009392 (0x4cef bytes) 000998 1788522932.891594 write 0x00dae5 thru 0x011d4b (0x4267 bytes) 001000 1788522932.896813 mapwrite 0x032765 thru 0x0367ac (0x4048 bytes) All operations completed A-OK! PASS 70a (123s) == sanity-flr test 71: check mirror extend parent fid ==== 07:55:38 (1788522938) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 07:55:41 (1788522941) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.285322 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13058 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 07:55:51 (1788522951) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6934240 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2943040 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7906048 .. Extending file size to 5048608 .. Extending file size to 8568832 .. Extending file size to 8751264 .. Extending file size to 6134688 .. Extending file size to 7189408 .. Extending file size to 9002016 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9200512 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 57306 57307 57308 57311 57313 failed 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) 192.168.206.116@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:57:16 (1788523036) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5319936 .. Extending file size to 3780256 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6914784 .. Extending file size to 8260864 .. Extending file size to 8564512 .. Extending file size to 6042048 .. Extending file size to 7636384 .. Extending file size to 7949280 .. Extending file size to 8325888 .. Extending file size to 8743168 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8616896 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8933344 .. Extending file size to 8701536 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58775 58776 58778 58779 58781 done 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) 192.168.206.116@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:58:38 (1788523118) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123105 s, 17.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.458664 s, 4.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x3001422d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:59:00 (1788523140) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192337 s, 10.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 07:59:08 (1788523148) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:59:17 (1788523157) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163554 s, 12.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:59:26 (1788523166) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.249027 s, 8.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:59:34 (1788523174) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.251146 s, 8.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:00:06 (1788523206) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg616-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104765 s, 10.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:01:03 (1788523263) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg616-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274429 s, 7.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:02:01 (1788523321) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg616-server mds-ost sync done. Creating new pool test_204f oleg616-server: Pool lustre.test_204f created Adding targets to pool oleg616-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg616-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.233982 s, 9.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg616-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg616-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg616-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:03:21 (1788523401) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:03:29 (1788523409) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:03:36 (1788523416) oleg616-server: Pool lustre.test_206 created Adding targets to pool oleg616-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg616-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg616-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg616-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg616-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:04:02 (1788523442) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572683 s, 18.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055305 s, 19.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7c9c0490 Mirror 3: 0x7c9c0490 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:04:13 (1788523453) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:04:16 (1788523456) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:04:20 (1788523460) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.170176 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xac0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x13e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x13e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x13e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x870000 Read this (4 bytes): BBBB 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:04:58 (1788523498) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0367953 s, 223 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:05:06 (1788523506) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425773 s, 24.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f8 lmm_fid: [0x200000404:0x1f8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f8 lmm_fid: [0x200000404:0x1f8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834424 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 08:05:07.000000000 -0400 Modify: 2026-09-04 08:05:07.000000000 -0400 Change: 2026-09-04 08:05:09.000000000 -0400 Birth: 2026-09-04 08:05:09.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:05:15 (1788523515) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:05:18 (1788523518) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.542605 s, 75.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.1383 s, 29.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 4 08:05 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fa:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 4 08:05 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:05:27 (1788523527) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:05:34 (1788523534) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1774 sec =========== 08:05:44 (1788523544) === sanity-flr: start cleanup 08:05:46 (1788523546) === === sanity-flr: finish cleanup 08:05:53 (1788523553) ===