-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 23:20:50 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg421-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 23:21:12 (1788837672) === oleg421-client.virtnet: executing check_config_client /mnt/lustre oleg421-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg421-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88c792832800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88c792832800.idle_timeout=debug disable quota as required oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 23:21:50 (1788837710) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7624] == sanity-flr test 0a: lfs mirror create with -N option == 23:21:53 (1788837713) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:22:15 (1788837735) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:22:19 (1788837739) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:22:23 (1788837743) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:22:51 (1788837771) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:22:55 (1788837775) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 23:22:59 (1788837779) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88c792832800.stats=clear osc.lustre-OST0001-osc-ffff88c792832800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:23:08 (1788837788) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:23:21 (1788837801) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:23:31 (1788837811) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:23:34 (1788837814) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:23:44 (1788837824) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 23:23:55 (1788837835) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:24:02 (1788837842) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:24:06 (1788837846) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:24:14 (1788837854) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:24:25 (1788837865) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:24:32 (1788837872) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 3.30111 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200368 s, 5.2 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:24:47 (1788837887) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.463026 s, 24.9 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:24:57 (1788837897) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:25:10 (1788837910) Stopping /mnt/lustre-ost1 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88c792832800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88c792832800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg421-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88c792832800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88c792832800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg421-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (79s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:26:30 (1788837990) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.396366 s, 10.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88c792832800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88c792832800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg421-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:27:17 (1788838037) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88c792832800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88c792832800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg421-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88c792832800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88c792832800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg421-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (90s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:28:47 (1788838127) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.61356 s, 9.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 23:28 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.114s user 0m0.000s sys 0m0.172s Stopping /mnt/lustre-ost1 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88c792832800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88c792832800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.366s user 0m0.000s sys 0m0.212s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg421-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg421-server oleg421-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88c792832800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88c792832800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.362s user 0m0.001s sys 0m0.204s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg421-server: oleg421-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg421-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg421-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88c792832800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88c792832800.ost_server_uuid in FULL state after 0 sec PASS 33b (99s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:30:27 (1788838227) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:30:30 (1788838230) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:30:34 (1788838234) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 23:30:37 (1788838237) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 23:30:45 (1788838245) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.712901 s, 17.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:30:59 (1788838259) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451487 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024412 s, 43.0 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:31:07 (1788838267) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:31:15 (1788838275) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00136944 s, 748 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127633 s, 802 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0102908 s, 99.5 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00181026 s, 566 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 23:31:26 (1788838286) create test files with size 15738917 .. 525714479793e0c79c5cf6b24dc766cb - ff89fe4ca522aa4dca0467eb38a91ca0 - fdf7fde3d670f2555a3d0195f19a4b28 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg421-client.virtnet /mnt/lustre (opts:) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 23:31:50 (1788838310) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:32:03 (1788838323) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:32:11 (1788838331) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153909 s, 6.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd3:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd4:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554227 s, 18.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd5:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 23:32:20 (1788838340) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131858 s, 15.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.400453 s, 10.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192691 s, 10.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 23:32:38 (1788838358) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 23:32:42 (1788838362) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:32:47 (1788838367) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561359 s, 18.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (12s) == sanity-flr test 44a: lfs mirror split check =========== 23:32:58 (1788838378) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:33:01 (1788838381) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (12s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:33:15 (1788838395) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.520906 s, 20.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg421-client.virtnet /mnt/lustre (opts:) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:33:31 (1788838411) llite.lustre-ffff88c794747000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.526837 s, 19.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg421-client.virtnet /mnt/lustre (opts:) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff88c7a0856000.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:33:51 (1788838431) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:34:35 (1788838475) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:35:16 (1788838516) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 23:35:20 (1788838520) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.350638 s, 9.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.176235 s, 17.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9f131d59 (nosync,stale) Mirror 2: 0x9f131d59 (nosync,stale) ** verify mirror 2 doesn't change original checksum: cd720a87fb1c18bf2cd122295a55037e - mirror 1 checksum: 13a78dff26c7e72ea26190e685ca1bb7 - mirror 2 checksum: cd720a87fb1c18bf2cd122295a55037e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T23:35:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T23:35:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: cd720a87fb1c18bf2cd122295a55037e - mirror 1 checksum: 13a78dff26c7e72ea26190e685ca1bb7 - mirror 2 checksum: 13a78dff26c7e72ea26190e685ca1bb7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T23:35:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T23:35:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:35:34 (1788838534) Failing mds1 on oleg421-server Stopping /mnt/lustre-mds1 (opts:) on oleg421-server 23:35:42 (1788838542) shut down facet: mds1 facet_host: oleg421-server facet_failover_host: oleg421-server Failover mds1 to oleg421-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 23:36:06 (1788838566) targets are mounted 23:36:06 (1788838566) facet_failover done oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275575 s, 38.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (50s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:36:24 (1788838584) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:36:29 (1788838589) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:36:32 (1788838592) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00669218 s, 2.4 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:36:40 (1788838600) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:36:44 (1788838604) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0150329 s, 272 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.202819 s, 20.2 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:36:54 (1788838614) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:37:20 (1788838640) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:37:51 (1788838671) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:38:15 (1788838695) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:38:41 (1788838721) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 23:38:50 (1788838730) mirror create pid 54352 mirror split pid 54354 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54352 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54354 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 23:39:58 (1788838798) Seed set to 8800 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000008 1788838801.180979 write 0x01c466 thru 0x025ddb (0x9976 bytes) 000009 1788838801.186846 mapwrite 0x029afc thru 0x0349a1 (0xaea6 bytes) 000010 1788838801.385595 mapread 0x0097a1 thru 0x0105af (0x6e0f bytes) 000012 1788838801.400843 write 0x012b87 thru 0x020d11 (0xe18b bytes) 000014 1788838801.408199 write 0x03ff29 thru 0x03ffff (0xd7 bytes) 000017 1788838801.411597 read 0x008410 thru 0x0144d9 (0xc0ca bytes) truncating to largest ever: 0x1d401 000018 1788838801.412334 trunc from 0x040000 to 0x01d400 (0x22bff bytes) 000019 1788838801.443539 mapread 0x00bc16 thru 0x016916 (0xad01 bytes) 000021 1788838801.453059 write 0x02cbe0 thru 0x02dea8 (0x12c9 bytes) truncating to largest ever: 0x25cec 000023 1788838801.454881 trunc from 0x02dea9 to 0x025ceb (0x81bd bytes) 000024 1788838801.479216 punch from 0x01e1e5 to 0x02354e (0x536a bytes) 000025 1788838801.505752 punch from 0x0014c2 to 0x008024 (0x6b63 bytes) 000029 1788838801.515918 mapread 0x019382 thru 0x01bafe (0x277d bytes) 000032 1788838801.519720 trunc from 0x025cec to 0x013bc3 (0x12128 bytes) 000034 1788838801.549857 write 0x0320c9 thru 0x03ffff (0xdf37 bytes) 000035 1788838801.556174 read 0x0258ea thru 0x02f162 (0x9879 bytes) 000036 1788838801.567756 write 0x002ba1 thru 0x0078ee (0x4d4e bytes) 000037 1788838801.568967 mapread 0x026f9c thru 0x02ef6b (0x7fd0 bytes) 000038 1788838801.571459 trunc from 0x040000 to 0x01f024 (0x20fdb bytes) 000040 1788838801.591572 mapwrite 0x038226 thru 0x03b1c0 (0x2f9b bytes) 000041 1788838801.680701 mapwrite 0x01a4c2 thru 0x01faab (0x55ea bytes) 000042 1788838801.727678 MIRROR_EXTEND to 2 mirrors 000043 1788838802.840109 write 0x008f34 thru 0x0174b5 (0xe582 bytes) 000045 1788838802.964294 mapwrite 0x02e891 thru 0x0385a9 (0x9d19 bytes) 000046 1788838803.127326 read 0x01fb19 thru 0x023f18 (0x4400 bytes) 000047 1788838803.139491 write 0x023720 thru 0x02a25e (0x6b3f bytes) 000049 1788838803.145046 read 0x005152 thru 0x012a49 (0xd8f8 bytes) 000051 1788838803.164481 punch from 0x024821 to 0x027b2c (0x330c bytes) 000052 1788838803.181501 mapread 0x01c27a thru 0x02018d (0x3f14 bytes) 000058 1788838803.187517 mapwrite 0x00e9f5 thru 0x01ceda (0xe4e6 bytes) 000059 1788838803.310275 read 0x01737d thru 0x01b1f0 (0x3e74 bytes) 000060 1788838803.318540 MIRROR_RESYNC 2 mirrors 000061 1788838804.200344 MIRROR_EXTEND to 3 mirrors truncating to largest ever: 0x3c8ef 000062 1788838804.917205 trunc from 0x03b1c1 to 0x03c8ee (0x172e bytes) 000063 1788838805.039848 punch from 0x022cc3 to 0x02a6aa (0x79e8 bytes) 000066 1788838805.052291 mapwrite 0x03aa90 thru 0x03cf43 (0x24b4 bytes) 000068 1788838805.201216 write 0x035993 thru 0x03b01f (0x568d bytes) 000070 1788838805.204100 read 0x02ca93 thru 0x03b0d3 (0xe641 bytes) 000071 1788838805.234178 MIRROR_RESYNC 3 mirrors 000072 1788838806.754332 trunc from 0x03cf44 to 0x01d04f (0x1fef4 bytes) 000074 1788838806.952732 trunc from 0x01d050 to 0x03106d (0x1401e bytes) 000077 1788838806.994851 MIRROR_RESYNC 3 mirrors 000081 1788838808.184048 mapwrite 0x007ca3 thru 0x008c6c (0xfca bytes) 000082 1788838808.583059 trunc from 0x03106e to 0x022a29 (0xe644 bytes) 000083 1788838808.615247 mapwrite 0x024259 thru 0x032c48 (0xe9f0 bytes) 000084 1788838808.819074 trunc from 0x032c49 to 0x033fbd (0x1375 bytes) 000085 1788838808.905254 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33fbe): Mirror 2: 0x4dd8776b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fbe): Mirror 2: 0x4dd8776b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fbe: skipped 000086 1788838809.376893 write 0x002682 thru 0x00813f (0x5abe bytes) 000087 1788838809.390956 mapwrite 0x023471 thru 0x02b596 (0x8126 bytes) 000088 1788838809.494126 write 0x011a0a thru 0x01217b (0x772 bytes) 000089 1788838809.506334 punch from 0x00ab8c to 0x00b4a6 (0x91b bytes) 000090 1788838809.515580 trunc from 0x033fbe to 0x013802 (0x207bb bytes) 000093 1788838809.543974 mapread 0x007707 thru 0x00df1d (0x6817 bytes) 000094 1788838809.557967 trunc from 0x013803 to 0x0062a4 (0xd55e bytes) 000095 1788838809.577369 write 0x021dfb thru 0x030acc (0xecd2 bytes) 000096 1788838809.580921 punch from 0x00544d to 0x00b5a6 (0x615a bytes) 000098 1788838809.600409 read 0x00c3d0 thru 0x017e2e (0xba5f bytes) 000099 1788838809.616433 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30acd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30acd): Mirror 2: 0x59005970 (stale) CRC-32 checksum value for chunk [0, 0x30acd): Mirror 1: 0x620d3be6 Mirror 3: 0x620d3be6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30acd: skipped 000100 1788838811.076008 MIRROR_RESYNC 3 mirrors 000101 1788838812.085271 trunc from 0x030acd to 0x003d56 (0x2cd76 bytes) 000103 1788838812.211516 write 0x0293cf thru 0x037a4c (0xe67e bytes) 000105 1788838812.215883 read 0x017625 thru 0x022b82 (0xb55e bytes) 000109 1788838812.256761 MIRROR_RESYNC 3 mirrors 000110 1788838813.347372 mapread 0x02eaba thru 0x0322ed (0x3834 bytes) 000111 1788838813.366007 trunc from 0x037a4d to 0x02454f (0x134fd bytes) 000113 1788838813.457464 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24550): Mirror 3: 0x5470339c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24550): Mirror 3: 0x5470339c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24550: skipped 000114 1788838813.885366 punch from 0x020e7f to 0x0232a1 (0x2423 bytes) 000115 1788838813.893116 punch from 0x000790 to 0x00d3e1 (0xcc52 bytes) 000116 1788838813.898697 punch from 0x015f34 to 0x01aa96 (0x4b63 bytes) 000117 1788838813.905027 mapwrite 0x0126b0 thru 0x0222c6 (0xfc17 bytes) 000118 1788838814.046260 trunc from 0x024550 to 0x017fd1 (0xc57e bytes) 000119 1788838814.084252 read 0x0104fe thru 0x017fd1 (0x7ad4 bytes) 000120 1788838814.085031 punch from 0x002441 to 0x0025b3 (0x173 bytes) 000121 1788838814.102972 MIRROR_RESYNC 2 mirrors 000124 1788838815.082281 read 0x00a16f thru 0x00eaaa (0x493c bytes) 000126 1788838815.092543 mapread 0x011508 thru 0x017fd1 (0x6aca bytes) 000130 1788838815.148843 mapread 0x006e39 thru 0x0150fc (0xe2c4 bytes) 000131 1788838815.182272 MIRROR_EXTEND to 3 mirrors 000132 1788838816.110635 read 0x00a4cc thru 0x011b56 (0x768b bytes) 000133 1788838816.149674 write 0x02f693 thru 0x0345c5 (0x4f33 bytes) 000134 1788838816.286924 write 0x01c2a6 thru 0x01e65c (0x23b7 bytes) 000135 1788838816.300479 mapread 0x01a676 thru 0x02866b (0xdff6 bytes) 000136 1788838816.349642 punch from 0x025496 to 0x0326e2 (0xd24d bytes) 000137 1788838816.361732 trunc from 0x0345c6 to 0x025c41 (0xe984 bytes) truncating to largest ever: 0x3f36f 000138 1788838816.420682 trunc from 0x025c42 to 0x03f36e (0x1972d bytes) 000142 1788838816.444491 mapread 0x0230ac thru 0x028f1d (0x5e72 bytes) 000143 1788838816.461960 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f36f): Mirror 4: 0x4e903cd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f36f): Mirror 4: 0x4e903cd0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f36f: skipped 000144 1788838816.848807 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f36f): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f36f): Mirror 4: 0x4e903cd0 (stale) CRC-32 checksum value for chunk [0, 0x3f36f): Mirror 1: 0xa15a7089 Mirror 5: 0xa15a7089 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f36f: skipped 000146 1788838818.255803 mapread 0x035822 thru 0x038a50 (0x322f bytes) 000147 1788838818.291889 write 0x034ce8 thru 0x039bc9 (0x4ee2 bytes) 000148 1788838818.379542 punch from 0x030eb7 to 0x032921 (0x1a6b bytes) 000149 1788838818.403201 write 0x0191ea thru 0x020fe2 (0x7df9 bytes) 000151 1788838818.418347 read 0x01b3c4 thru 0x023c1f (0x885c bytes) 000152 1788838818.427842 mapread 0x01a1fa thru 0x029d69 (0xfb70 bytes) 000153 1788838818.440228 read 0x03700b thru 0x03ab3c (0x3b32 bytes) 000154 1788838818.456888 trunc from 0x03f36f to 0x022780 (0x1cbee bytes) 000156 1788838818.481178 read 0x00a186 thru 0x012392 (0x820d bytes) 000157 1788838818.482230 write 0x035df7 thru 0x03ef38 (0x9142 bytes) 000160 1788838818.489529 read 0x00cf6e thru 0x0168d6 (0x9969 bytes) 000162 1788838818.490948 trunc from 0x03ef39 to 0x0130d9 (0x2be5f bytes) 000163 1788838818.515089 trunc from 0x0130da to 0x011c76 (0x1463 bytes) 000164 1788838818.535901 write 0x01e5f0 thru 0x026ac0 (0x84d1 bytes) 000168 1788838818.542145 mapread 0x00c984 thru 0x018409 (0xba86 bytes) 000169 1788838818.556157 mapread 0x01bb21 thru 0x022a56 (0x6f36 bytes) 000170 1788838818.557307 read 0x000526 thru 0x002c5b (0x2736 bytes) 000171 1788838818.559007 mapwrite 0x025676 thru 0x0352d5 (0xfc60 bytes) 000173 1788838818.691727 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x352d6): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x352d6): Mirror 4: 0x32eb1406 (stale) Mirror 5: 0xf3d8b924 (stale) CRC-32 checksum value for chunk [0, 0x352d6): Mirror 1: 0x1ed7224a Mirror 6: 0x1ed7224a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x352d6: skipped 000175 1788838819.982469 read 0x02c8fc thru 0x031210 (0x4915 bytes) 000176 1788838820.003265 mapread 0x007f1a thru 0x00f31d (0x7404 bytes) 000177 1788838820.052960 trunc from 0x0352d6 to 0x013bbe (0x21717 bytes) 000178 1788838820.161843 punch from 0x00c34b to 0x00f5c5 (0x327b bytes) 000179 1788838820.172280 mapread 0x00ef89 thru 0x00f863 (0x8db bytes) 000180 1788838820.187725 trunc from 0x013bbf to 0x02dc32 (0x1a074 bytes) 000181 1788838820.238822 trunc from 0x02dc33 to 0x004b3f (0x290f3 bytes) 000182 1788838820.279104 mapread 0x003725 thru 0x004b3f (0x141b bytes) 000183 1788838820.299180 mapwrite 0x004baa thru 0x0079f1 (0x2e48 bytes) 000185 1788838820.419118 mapwrite 0x030519 thru 0x03af19 (0xaa01 bytes) 000186 1788838820.540139 trunc from 0x03af1a to 0x02b9c6 (0xf553 bytes) 000188 1788838820.573084 write 0x023bf8 thru 0x029f36 (0x633f bytes) 000189 1788838820.589043 mapwrite 0x016ba9 thru 0x025b95 (0xefed bytes) 000191 1788838820.680539 MIRROR_RESYNC 4 mirrors 000193 1788838821.618834 write 0x02089f thru 0x023a01 (0x3163 bytes) 000197 1788838821.775621 write 0x016bf5 thru 0x024106 (0xd512 bytes) 000198 1788838821.806487 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2b9c7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b9c7): Mirror 4: 0x59196cd9 (stale) Mirror 5: 0x59196cd9 (stale) Mirror 6: 0x59196cd9 (stale) CRC-32 checksum value for chunk [0, 0x2b9c7): Mirror 1: 0x60e031a7 Mirror 7: 0x60e031a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b9c7: skipped 000199 1788838823.219534 read 0x02296e thru 0x02b9c6 (0x9059 bytes) 000200 1788838823.246132 read 0x026bdc thru 0x02b972 (0x4d97 bytes) 000201 1788838823.261318 mapwrite 0x01319a thru 0x0132ea (0x151 bytes) 000203 1788838823.555175 write 0x001f45 thru 0x003ce2 (0x1d9e bytes) 000205 1788838823.593724 mapwrite 0x02d4d3 thru 0x03392b (0x6459 bytes) 000206 1788838823.735342 read 0x00c7ec thru 0x011965 (0x517a bytes) 000207 1788838823.762719 write 0x02b89c thru 0x03a360 (0xeac5 bytes) 000208 1788838823.768928 mapwrite 0x0110e4 thru 0x01eb19 (0xda36 bytes) 000210 1788838823.912263 read 0x0245c4 thru 0x02f631 (0xb06e bytes) 000211 1788838823.917732 write 0x012fea thru 0x0170b1 (0x40c8 bytes) 000212 1788838823.936525 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a361): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a361): Mirror 4: 0x181427ae (stale) Mirror 5: 0x181427ae (stale) Mirror 6: 0x181427ae (stale) Mirror 7: 0x9bf38f7 (stale) CRC-32 checksum value for chunk [0, 0x3a361): Mirror 1: 0x48a9b4e8 Mirror 8: 0x48a9b4e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a361: skipped 000214 1788838825.217581 read 0x026ebd thru 0x02e9f0 (0x7b34 bytes) 000215 1788838825.243680 punch from 0x015799 to 0x01cafc (0x7364 bytes) 000216 1788838825.352325 mapread 0x01abaa thru 0x028d45 (0xe19c bytes) 000217 1788838825.408083 punch from 0x037ce2 to 0x0383cc (0x6eb bytes) 000218 1788838825.420990 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a361): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a361): Mirror 4: 0x181427ae (stale) Mirror 5: 0x181427ae (stale) Mirror 6: 0x181427ae (stale) Mirror 7: 0x9bf38f7 (stale) Mirror 8: 0x48a9b4e8 (stale) CRC-32 checksum value for chunk [0, 0x3a361): Mirror 1: 0x5eb0ec99 Mirror 9: 0x5eb0ec99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a361: skipped 000221 1788838826.733475 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a361): Mirror 4: 0x181427ae (stale) Mirror 5: 0x181427ae (stale) Mirror 6: 0x181427ae (stale) Mirror 8: 0x48a9b4e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a361): Mirror 4: 0x181427ae (stale) Mirror 5: 0x181427ae (stale) Mirror 6: 0x181427ae (stale) Mirror 8: 0x48a9b4e8 (stale) CRC-32 checksum value for chunk [0, 0x3a361): Mirror 1: 0x5eb0ec99 Mirror 9: 0x5eb0ec99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a361: skipped 000222 1788838827.643175 trunc from 0x03a361 to 0x0196b5 (0x20cab bytes) 000223 1788838827.749371 MIRROR_RESYNC 6 mirrors 000224 1788838828.988667 read 0x0149eb thru 0x0196b5 (0x4ccb bytes) 000225 1788838829.011132 MIRROR_RESYNC 6 mirrors 000227 1788838829.662914 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000229 1788838830.075961 read 0x014bb5 thru 0x016b33 (0x1f7f bytes) 000230 1788838830.090989 write 0x01b52e thru 0x01bc40 (0x713 bytes) 000231 1788838830.183908 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1bc41): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bc41): Mirror 5: 0x79dbee70 (stale) Mirror 6: 0x79dbee70 (stale) Mirror 8: 0x79dbee70 (stale) Mirror 9: 0x79dbee70 (stale) CRC-32 checksum value for chunk [0, 0x1bc41): Mirror 1: 0x45e07d7c Mirror 10: 0x45e07d7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bc41: skipped 000232 1788838831.311959 trunc from 0x01bc41 to 0x03a64f (0x1ea0f bytes) 000233 1788838831.436605 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a650): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a650): Mirror 5: 0xe58b3984 (stale) Mirror 6: 0xe58b3984 (stale) Mirror 8: 0xe58b3984 (stale) Mirror 9: 0xe58b3984 (stale) Mirror 10: 0x3c130c69 (stale) CRC-32 checksum value for chunk [0, 0x3a650): Mirror 1: 0x3c130c69 Mirror 11: 0x3c130c69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a650: skipped 000234 1788838832.767848 trunc from 0x03a650 to 0x03b66f (0x1020 bytes) 000235 1788838832.892355 read 0x00b746 thru 0x01565b (0x9f16 bytes) 000236 1788838832.906609 MIRROR_RESYNC 7 mirrors 000237 1788838834.906660 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000238 1788838835.348889 trunc from 0x03b670 to 0x032409 (0x9266 bytes) 000239 1788838835.462997 trunc from 0x03240a to 0x00d352 (0x250b7 bytes) 000240 1788838835.508639 mapread 0x000f75 thru 0x00d352 (0xc3de bytes) 000241 1788838835.559764 mapread 0x005b9b thru 0x00d352 (0x77b8 bytes) 000242 1788838835.563927 mapwrite 0x03416c thru 0x03fe23 (0xbcb8 bytes) 000243 1788838835.709874 write 0x019346 thru 0x01c2df (0x2f9a bytes) 000245 1788838835.723885 write 0x037c81 thru 0x03b70c (0x3a8c bytes) 000247 1788838835.727916 mapwrite 0x02bb10 thru 0x02c66f (0xb60 bytes) 000249 1788838835.789875 trunc from 0x03fe24 to 0x02a0b1 (0x15d72 bytes) 000250 1788838835.809908 punch from 0x00a3c2 to 0x0190fa (0xed39 bytes) 000251 1788838835.819902 mapread 0x027bc2 thru 0x029eab (0x22ea bytes) 000252 1788838835.821063 trunc from 0x02a0b2 to 0x03b6a8 (0x115f7 bytes) 000253 1788838835.847373 write 0x03edc4 thru 0x03ffff (0x123c bytes) 000254 1788838835.848684 trunc from 0x040000 to 0x015efa (0x2a105 bytes) 000257 1788838835.876281 trunc from 0x015efb to 0x011d27 (0x41d3 bytes) 000258 1788838835.905347 read 0x00f04b thru 0x011d27 (0x2cdd bytes) 000259 1788838835.920080 read 0x003668 thru 0x0087ad (0x5146 bytes) 000260 1788838835.922192 read 0x008908 thru 0x01019c (0x7895 bytes) 000263 1788838835.936811 write 0x022dd2 thru 0x02d363 (0xa592 bytes) 000265 1788838835.941572 mapread 0x0134d7 thru 0x01dd81 (0xa8ab bytes) 000267 1788838835.951879 mapread 0x012593 thru 0x0173f8 (0x4e66 bytes) 000269 1788838835.954817 write 0x02916c thru 0x02a208 (0x109d bytes) 000271 1788838835.958290 punch from 0x002eaa to 0x00c9b6 (0x9b0d bytes) 000273 1788838835.975927 write 0x015180 thru 0x01667a (0x14fb bytes) 000274 1788838835.977024 read 0x01675b thru 0x016918 (0x1be bytes) 000275 1788838835.982606 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2d364): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d364): Mirror 6: 0x8a5400fd (stale) Mirror 8: 0x8a5400fd (stale) Mirror 9: 0x8a5400fd (stale) Mirror 10: 0x8a5400fd (stale) Mirror 11: 0x8a5400fd (stale) CRC-32 checksum value for chunk [0, 0x2d364): Mirror 1: 0x6dac109b Mirror 12: 0x6dac109b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d364: skipped 000277 1788838837.128062 mapread 0x02a662 thru 0x02d363 (0x2d02 bytes) 000279 1788838837.185100 mapread 0x016375 thru 0x01a627 (0x42b3 bytes) 000280 1788838837.229998 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2d364): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d364): Mirror 6: 0x8a5400fd (stale) Mirror 8: 0x8a5400fd (stale) Mirror 9: 0x8a5400fd (stale) Mirror 10: 0x8a5400fd (stale) Mirror 11: 0x8a5400fd (stale) CRC-32 checksum value for chunk [0, 0x2d364): Mirror 1: 0x6dac109b Mirror 12: 0x6dac109b Mirror 13: 0x6dac109b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d364: skipped 000281 1788838838.484056 punch from 0x01fca1 to 0x020a87 (0xde7 bytes) 000283 1788838838.587636 trunc from 0x02d364 to 0x0200e7 (0xd27c bytes) 000284 1788838838.634029 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x200e8): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x200e8): Mirror 6: 0x20040f47 (stale) Mirror 8: 0x20040f47 (stale) Mirror 9: 0x20040f47 (stale) Mirror 10: 0x20040f47 (stale) Mirror 11: 0x20040f47 (stale) Mirror 12: 0x563f6fcf (stale) Mirror 13: 0x563f6fcf (stale) CRC-32 checksum value for chunk [0, 0x200e8): Mirror 1: 0x563f6fcf Mirror 14: 0x563f6fcf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x200e8: skipped 000287 1788838840.085108 mapwrite 0x00aa34 thru 0x00b189 (0x756 bytes) 000288 1788838840.359414 read 0x008891 thru 0x010d18 (0x8488 bytes) 000289 1788838840.360561 mapread 0x01a3b6 thru 0x0200e7 (0x5d32 bytes) 000291 1788838840.381892 MIRROR_RESYNC 9 mirrors 000292 1788838842.520150 mapwrite 0x02fe1e thru 0x033a0f (0x3bf2 bytes) 000295 1788838842.826572 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33a10): Mirror 6: 0x7366773 (stale) Mirror 8: 0x7366773 (stale) Mirror 9: 0x7366773 (stale) Mirror 11: 0x7366773 (stale) Mirror 12: 0x7366773 (stale) Mirror 13: 0x7366773 (stale) Mirror 14: 0x7366773 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33a10): Mirror 6: 0x7366773 (stale) Mirror 8: 0x7366773 (stale) Mirror 9: 0x7366773 (stale) Mirror 11: 0x7366773 (stale) Mirror 12: 0x7366773 (stale) Mirror 13: 0x7366773 (stale) Mirror 14: 0x7366773 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33a10: skipped 000296 1788838843.370477 read 0x025b33 thru 0x02aaea (0x4fb8 bytes) 000297 1788838843.384917 mapwrite 0x032730 thru 0x03ffff (0xd8d0 bytes) 000298 1788838843.538062 mapread 0x01f12b thru 0x021752 (0x2628 bytes) 000299 1788838843.555366 punch from 0x03e9fb to 0x03ffff (0x1605 bytes) 000300 1788838843.584715 mapwrite 0x009db9 thru 0x009deb (0x33 bytes) 000303 1788838843.688048 mapread 0x021e80 thru 0x026b5c (0x4cdd bytes) 000304 1788838843.703510 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1a373e6a (stale) Mirror 8: 0x1a373e6a (stale) Mirror 9: 0x1a373e6a (stale) Mirror 11: 0x1a373e6a (stale) Mirror 12: 0x1a373e6a (stale) Mirror 13: 0x1a373e6a (stale) Mirror 14: 0x1a373e6a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x942a6443 Mirror 15: 0x942a6443 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000305 1788838845.187368 read 0x01de17 thru 0x01f0f0 (0x12da bytes) 000306 1788838845.251862 MIRROR_RESYNC 9 mirrors 000307 1788838847.170946 trunc from 0x040000 to 0x01c0c5 (0x23f3a bytes) 000308 1788838847.286003 mapread 0x01365f thru 0x017e90 (0x4832 bytes) 000309 1788838847.322694 punch from 0x00758e to 0x0132f8 (0xbd6b bytes) 000311 1788838847.328843 trunc from 0x01c0c6 to 0x0397eb (0x1d726 bytes) 000314 1788838847.354124 read 0x0218a1 thru 0x02bcab (0xa40b bytes) 000315 1788838847.367977 mapread 0x01d324 thru 0x022bf8 (0x58d5 bytes) 000316 1788838847.378446 read 0x0343ee thru 0x036ac1 (0x26d4 bytes) 000317 1788838847.390297 mapwrite 0x00a548 thru 0x0180eb (0xdba4 bytes) 000320 1788838847.473990 trunc from 0x0397ec to 0x0153ca (0x24421 bytes) 000321 1788838847.527129 MIRROR_RESYNC 9 mirrors 000323 1788838849.807784 write 0x03c127 thru 0x03ffff (0x3ed9 bytes) truncating to largest ever: 0x3fcb5 000325 1788838849.934262 trunc from 0x040000 to 0x03fcb4 (0x34b bytes) 000326 1788838849.973014 punch from 0x02298d to 0x02f37e (0xc9f2 bytes) 000327 1788838849.988193 mapread 0x0330b9 thru 0x039a4c (0x6994 bytes) 000329 1788838850.016505 read 0x03cc10 thru 0x03fcb4 (0x30a5 bytes) 000330 1788838850.017078 trunc from 0x03fcb5 to 0x0377b2 (0x8502 bytes) 000334 1788838850.049362 read 0x02d92f thru 0x02fc4b (0x231d bytes) 000335 1788838850.074987 punch from 0x01b5d0 to 0x01ccaf (0x16e0 bytes) 000337 1788838850.086953 punch from 0x00b0aa to 0x015d79 (0xacd0 bytes) 000338 1788838850.091301 trunc from 0x0377b3 to 0x01d5d3 (0x1a1df bytes) 000339 1788838850.144250 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1d5d4): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d5d4): Mirror 6: 0x78d5ee3f (stale) Mirror 8: 0x78d5ee3f (stale) Mirror 9: 0x78d5ee3f (stale) Mirror 11: 0x78d5ee3f (stale) Mirror 12: 0x78d5ee3f (stale) Mirror 13: 0x78d5ee3f (stale) Mirror 14: 0x78d5ee3f (stale) Mirror 15: 0x78d5ee3f (stale) CRC-32 checksum value for chunk [0, 0x1d5d4): Mirror 1: 0x20cabac2 Mirror 16: 0x20cabac2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d5d4: skipped 000341 1788838851.554062 write 0x03c302 thru 0x03ffff (0x3cfe bytes) 000342 1788838851.678580 punch from 0x038d79 to 0x03ffff (0x7287 bytes) 000343 1788838851.690243 trunc from 0x040000 to 0x01cff4 (0x2300b bytes) 000345 1788838851.755738 mapread 0x01799f thru 0x01cff4 (0x5656 bytes) 000347 1788838851.782869 trunc from 0x01cff5 to 0x00d368 (0xfc8c bytes) 000348 1788838851.811956 mapread 0x00b25e thru 0x00d368 (0x210b bytes) 000349 1788838851.838095 punch from 0x0071f3 to 0x00d368 (0x6176 bytes) 000350 1788838851.850110 trunc from 0x00d369 to 0x006d0b (0x665d bytes) 000351 1788838851.884839 punch from 0x0023e0 to 0x006d0b (0x492c bytes) 000352 1788838851.892255 trunc from 0x006d0c to 0x039232 (0x32527 bytes) 000354 1788838851.916669 trunc from 0x039233 to 0x03f901 (0x66cf bytes) 000355 1788838851.944251 mapwrite 0x01a0a2 thru 0x028e3e (0xed9d bytes) 000358 1788838852.041018 read 0x024d49 thru 0x02e294 (0x954c bytes) 000359 1788838852.046802 mapread 0x00bdd3 thru 0x014a59 (0x8c87 bytes) 000364 1788838852.062161 punch from 0x007d0f to 0x010ef0 (0x91e2 bytes) 000365 1788838852.069066 mapwrite 0x0338c7 thru 0x037c94 (0x43ce bytes) 000366 1788838852.162565 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f902): Mirror 6: 0x23deb5e8 (stale) Mirror 8: 0x23deb5e8 (stale) Mirror 9: 0x23deb5e8 (stale) Mirror 11: 0x23deb5e8 (stale) Mirror 12: 0x23deb5e8 (stale) Mirror 13: 0x23deb5e8 (stale) Mirror 15: 0x23deb5e8 (stale) Mirror 16: 0x49166257 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f902): Mirror 6: 0x23deb5e8 (stale) Mirror 8: 0x23deb5e8 (stale) Mirror 9: 0x23deb5e8 (stale) Mirror 11: 0x23deb5e8 (stale) Mirror 12: 0x23deb5e8 (stale) Mirror 13: 0x23deb5e8 (stale) Mirror 15: 0x23deb5e8 (stale) Mirror 16: 0x49166257 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f902: skipped 000367 1788838852.676099 punch from 0x02f7c2 to 0x039bd7 (0xa416 bytes) 000370 1788838852.685902 mapread 0x03793e thru 0x03f901 (0x7fc4 bytes) 000372 1788838852.725048 punch from 0x0277b9 to 0x033bba (0xc402 bytes) 000373 1788838852.738243 trunc from 0x03f902 to 0x0084c4 (0x3743d bytes) 000374 1788838852.802024 write 0x020496 thru 0x02f638 (0xf1a3 bytes) 000375 1788838852.810785 MIRROR_RESYNC 9 mirrors 000376 1788838855.112139 trunc from 0x02f639 to 0x02250c (0xd12c bytes) 000377 1788838855.206199 read 0x020df2 thru 0x02250c (0x171b bytes) 000378 1788838855.216114 punch from 0x0104f7 to 0x01e53b (0xe045 bytes) 000379 1788838855.222536 trunc from 0x02250d to 0x02a319 (0x7e0d bytes) 000380 1788838855.246088 write 0x00e439 thru 0x015a93 (0x765b bytes) 000382 1788838855.258782 punch from 0x009b8e to 0x017563 (0xd9d6 bytes) 000383 1788838855.279989 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2a31a): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a31a): Mirror 6: 0xb37c5b36 (stale) Mirror 8: 0xb37c5b36 (stale) Mirror 9: 0xb37c5b36 (stale) Mirror 11: 0xb37c5b36 (stale) Mirror 12: 0xb37c5b36 (stale) Mirror 13: 0xb37c5b36 (stale) Mirror 15: 0xb37c5b36 (stale) Mirror 16: 0xb37c5b36 (stale) CRC-32 checksum value for chunk [0, 0x2a31a): Mirror 1: 0xc8ff3f74 Mirror 17: 0xc8ff3f74 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a31a: skipped 000384 1788838856.600725 punch from 0x000f46 to 0x0097d1 (0x888c bytes) 000385 1788838856.706125 write 0x03d2bd thru 0x03ffff (0x2d43 bytes) 000386 1788838856.716433 MIRROR_RESYNC 10 mirrors 000387 1788838858.648697 punch from 0x01711b to 0x023289 (0xc16f bytes) 000388 1788838858.783290 write 0x007a80 thru 0x012b03 (0xb084 bytes) 000389 1788838858.796604 trunc from 0x040000 to 0x00b475 (0x34b8a bytes) 000390 1788838858.827000 mapwrite 0x01e55f thru 0x026fdc (0x8a7e bytes) 000391 1788838858.941050 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x26fdd): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26fdd): Mirror 6: 0x212a8111 (stale) Mirror 8: 0x212a8111 (stale) Mirror 9: 0x212a8111 (stale) Mirror 11: 0x212a8111 (stale) Mirror 12: 0x212a8111 (stale) Mirror 13: 0x212a8111 (stale) Mirror 15: 0x212a8111 (stale) Mirror 16: 0x212a8111 (stale) Mirror 17: 0x212a8111 (stale) CRC-32 checksum value for chunk [0, 0x26fdd): Mirror 1: 0x5afc05f8 Mirror 18: 0x5afc05f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26fdd: skipped 000393 1788838860.184491 mapread 0x001474 thru 0x00d4b8 (0xc045 bytes) 000395 1788838860.225917 mapwrite 0x004ec4 thru 0x0086f8 (0x3835 bytes) 000396 1788838860.398553 punch from 0x01da27 to 0x026a56 (0x9030 bytes) 000397 1788838860.410168 MIRROR_RESYNC 11 mirrors 000398 1788838862.381320 write 0x01d4be thru 0x02a386 (0xcec9 bytes) 000399 1788838862.487290 trunc from 0x02a387 to 0x02e104 (0x3d7e bytes) 000401 1788838862.519665 write 0x0260e3 thru 0x02ae54 (0x4d72 bytes) 000402 1788838862.522408 mapread 0x0042f5 thru 0x00b2f9 (0x7005 bytes) 000403 1788838862.576123 trunc from 0x02e105 to 0x00f69b (0x1ea69 bytes) 000404 1788838862.615217 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf69c): Mirror 8: 0xb8804564 (stale) Mirror 9: 0xb8804564 (stale) Mirror 11: 0xb8804564 (stale) Mirror 12: 0xb8804564 (stale) Mirror 13: 0xb8804564 (stale) Mirror 15: 0xb8804564 (stale) Mirror 16: 0xb8804564 (stale) Mirror 17: 0xb8804564 (stale) Mirror 18: 0xb8804564 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf69c): Mirror 8: 0xb8804564 (stale) Mirror 9: 0xb8804564 (stale) Mirror 11: 0xb8804564 (stale) Mirror 12: 0xb8804564 (stale) Mirror 13: 0xb8804564 (stale) Mirror 15: 0xb8804564 (stale) Mirror 16: 0xb8804564 (stale) Mirror 17: 0xb8804564 (stale) Mirror 18: 0xb8804564 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf69c: skipped 000405 1788838863.498167 write 0x013b9a thru 0x01f30a (0xb771 bytes) 000406 1788838863.499868 read 0x010802 thru 0x018a8a (0x8289 bytes) 000407 1788838863.507362 write 0x023c2d thru 0x02d0b7 (0x948b bytes) 000408 1788838863.516725 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2d0b8): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0b8): Mirror 8: 0x38b8934a (stale) Mirror 9: 0x38b8934a (stale) Mirror 11: 0x38b8934a (stale) Mirror 12: 0x38b8934a (stale) Mirror 13: 0x38b8934a (stale) Mirror 15: 0x38b8934a (stale) Mirror 16: 0x38b8934a (stale) Mirror 17: 0x38b8934a (stale) Mirror 18: 0x38b8934a (stale) CRC-32 checksum value for chunk [0, 0x2d0b8): Mirror 1: 0x813b0211 Mirror 19: 0x813b0211 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0b8: skipped 000410 1788838864.653643 mapread 0x005924 thru 0x01449a (0xeb77 bytes) 000412 1788838864.693103 write 0x009072 thru 0x009090 (0x1f bytes) 000413 1788838864.803823 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2d0b8): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0b8): Mirror 8: 0x38b8934a (stale) Mirror 9: 0x38b8934a (stale) Mirror 11: 0x38b8934a (stale) Mirror 12: 0x38b8934a (stale) Mirror 13: 0x38b8934a (stale) Mirror 15: 0x38b8934a (stale) Mirror 16: 0x38b8934a (stale) Mirror 17: 0x38b8934a (stale) Mirror 18: 0x38b8934a (stale) Mirror 19: 0x813b0211 (stale) CRC-32 checksum value for chunk [0, 0x2d0b8): Mirror 1: 0x5eea8e0f Mirror 20: 0x5eea8e0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0b8: skipped 000414 1788838865.954026 punch from 0x00f95f to 0x01c7ec (0xce8e bytes) 000416 1788838866.083838 write 0x01ba65 thru 0x02b7cf (0xfd6b bytes) 000418 1788838866.106087 MIRROR_RESYNC 12 mirrors 000420 1788838868.775140 mapread 0x0186a4 thru 0x026bf0 (0xe54d bytes) 000421 1788838868.821137 punch from 0x01a6a9 to 0x01b06a (0x9c2 bytes) 000422 1788838868.952214 mapread 0x003dec thru 0x0129f0 (0xec05 bytes) 000423 1788838868.981121 write 0x028338 thru 0x030b55 (0x881e bytes) 000424 1788838868.989191 read 0x012b7a thru 0x01c273 (0x96fa bytes) 000426 1788838869.002238 mapread 0x013a36 thru 0x018321 (0x48ec bytes) 000427 1788838869.013620 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x30b56): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b56): Mirror 8: 0x4f97b2bd (stale) Mirror 9: 0x4f97b2bd (stale) Mirror 11: 0x4f97b2bd (stale) Mirror 12: 0x4f97b2bd (stale) Mirror 13: 0x4f97b2bd (stale) Mirror 15: 0x4f97b2bd (stale) Mirror 16: 0x4f97b2bd (stale) Mirror 17: 0x4f97b2bd (stale) Mirror 18: 0x4f97b2bd (stale) Mirror 19: 0x4f97b2bd (stale) Mirror 20: 0x4f97b2bd (stale) CRC-32 checksum value for chunk [0, 0x30b56): Mirror 1: 0xb65fc1cc Mirror 21: 0xb65fc1cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b56: skipped 000428 1788838870.603057 trunc from 0x030b56 to 0x03994a (0x8df5 bytes) 000429 1788838870.720270 punch from 0x01d302 to 0x022f3c (0x5c3b bytes) 000430 1788838870.727262 trunc from 0x03994b to 0x0319e0 (0x7f6a bytes) 000431 1788838870.748754 read 0x02cb9d thru 0x02e464 (0x18c8 bytes) 000433 1788838870.756336 trunc from 0x0319e1 to 0x009eb3 (0x27b2d bytes) 000434 1788838870.772503 write 0x00fd23 thru 0x01b279 (0xb557 bytes) 000437 1788838870.781374 mapread 0x00d32f thru 0x01b07b (0xdd4d bytes) 000438 1788838870.791917 trunc from 0x01b27a to 0x008c4e (0x1262b bytes) 000440 1788838870.831029 MIRROR_RESYNC 13 mirrors 000441 1788838873.439219 mapread 0x0025a2 thru 0x008c4e (0x66ad bytes) 000442 1788838873.460487 mapwrite 0x008dbd thru 0x00e672 (0x58b6 bytes) 000443 1788838873.732840 mapwrite 0x0308dc thru 0x0363d6 (0x5afb bytes) 000445 1788838873.855456 mapread 0x01de86 thru 0x029aa9 (0xbc24 bytes) 000446 1788838873.878156 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x363d7): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x363d7): Mirror 8: 0xd00f1f7c (stale) Mirror 9: 0xd00f1f7c (stale) Mirror 11: 0xd00f1f7c (stale) Mirror 12: 0xd00f1f7c (stale) Mirror 13: 0xd00f1f7c (stale) Mirror 15: 0xd00f1f7c (stale) Mirror 16: 0xd00f1f7c (stale) Mirror 17: 0xd00f1f7c (stale) Mirror 18: 0xd00f1f7c (stale) Mirror 19: 0xd00f1f7c (stale) Mirror 20: 0xd00f1f7c (stale) Mirror 21: 0xd00f1f7c (stale) CRC-32 checksum value for chunk [0, 0x363d7): Mirror 1: 0xac908311 Mirror 22: 0xac908311 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x363d7: skipped 000449 1788838875.357003 mapwrite 0x014b76 thru 0x01f09d (0xa528 bytes) 000451 1788838875.656583 mapread 0x001ac9 thru 0x00e21c (0xc754 bytes) 000452 1788838875.679408 trunc from 0x0363d7 to 0x024cde (0x116f8 bytes) 000453 1788838875.720720 MIRROR_RESYNC 14 mirrors 000454 1788838878.961515 mapread 0x009804 thru 0x00e757 (0x4f54 bytes) 000455 1788838878.986626 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000456 1788838879.572603 trunc from 0x024cdf to 0x02f5ee (0xa910 bytes) 000457 1788838879.688153 read 0x00baef thru 0x01a866 (0xed78 bytes) 000458 1788838879.707542 MIRROR_RESYNC 13 mirrors 000460 1788838882.521586 write 0x032b2f thru 0x03fc86 (0xd158 bytes) 000462 1788838882.659040 trunc from 0x03fc87 to 0x022b8e (0x1d0f8 bytes) 000466 1788838882.699682 punch from 0x00d3ac to 0x01bd2b (0xe980 bytes) 000467 1788838882.709910 write 0x03fc41 thru 0x03ffff (0x3bf bytes) 000468 1788838882.712070 mapread 0x032f78 thru 0x03ffff (0xd088 bytes) 000470 1788838882.773663 write 0x03d196 thru 0x03ffff (0x2e6a bytes) 000471 1788838882.783973 MIRROR_SPLIT mirror 5 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xa6118cd5 (stale) Mirror 11: 0xa6118cd5 (stale) Mirror 12: 0xa6118cd5 (stale) Mirror 13: 0xa6118cd5 (stale) Mirror 16: 0xa6118cd5 (stale) Mirror 17: 0xa6118cd5 (stale) Mirror 18: 0xa6118cd5 (stale) Mirror 19: 0xa6118cd5 (stale) Mirror 20: 0xa6118cd5 (stale) Mirror 21: 0xa6118cd5 (stale) Mirror 22: 0xa6118cd5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xa6118cd5 (stale) Mirror 11: 0xa6118cd5 (stale) Mirror 12: 0xa6118cd5 (stale) Mirror 13: 0xa6118cd5 (stale) Mirror 16: 0xa6118cd5 (stale) Mirror 17: 0xa6118cd5 (stale) Mirror 18: 0xa6118cd5 (stale) Mirror 19: 0xa6118cd5 (stale) Mirror 20: 0xa6118cd5 (stale) Mirror 21: 0xa6118cd5 (stale) Mirror 22: 0xa6118cd5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000472 1788838883.880502 mapwrite 0x024515 thru 0x025926 (0x1412 bytes) 000473 1788838883.977916 write 0x03ad29 thru 0x03fa10 (0x4ce8 bytes) 000474 1788838883.990573 read 0x03dfce thru 0x03ffff (0x2032 bytes) 000475 1788838883.992128 mapwrite 0x01a467 thru 0x02913a (0xecd4 bytes) 000478 1788838884.070899 mapwrite 0x00265a thru 0x009490 (0x6e37 bytes) 000479 1788838884.141470 mapread 0x0104f2 thru 0x012cbe (0x27cd bytes) 000482 1788838884.149622 mapread 0x004676 thru 0x00bafb (0x7486 bytes) 000483 1788838884.151101 mapread 0x00d3a9 thru 0x0176cc (0xa324 bytes) 000485 1788838884.153560 mapwrite 0x01de36 thru 0x025346 (0x7511 bytes) 000486 1788838884.297624 mapread 0x0080ee thru 0x009d92 (0x1ca5 bytes) 000487 1788838884.298960 mapwrite 0x02e71a thru 0x03867a (0x9f61 bytes) 000488 1788838884.507233 read 0x003bca thru 0x0137b1 (0xfbe8 bytes) 000490 1788838884.510870 trunc from 0x040000 to 0x00c713 (0x338ec bytes) 000493 1788838884.540899 mapread 0x0067c0 thru 0x00c713 (0x5f54 bytes) 000494 1788838884.550623 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc714): Mirror 9: 0x4a6df067 (stale) Mirror 11: 0x4a6df067 (stale) Mirror 12: 0x4a6df067 (stale) Mirror 13: 0x4a6df067 (stale) Mirror 16: 0x4a6df067 (stale) Mirror 17: 0x4a6df067 (stale) Mirror 18: 0x4a6df067 (stale) Mirror 19: 0x4a6df067 (stale) Mirror 20: 0x4a6df067 (stale) Mirror 22: 0x4a6df067 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc714): Mirror 9: 0x4a6df067 (stale) Mirror 11: 0x4a6df067 (stale) Mirror 12: 0x4a6df067 (stale) Mirror 13: 0x4a6df067 (stale) Mirror 16: 0x4a6df067 (stale) Mirror 17: 0x4a6df067 (stale) Mirror 18: 0x4a6df067 (stale) Mirror 19: 0x4a6df067 (stale) Mirror 20: 0x4a6df067 (stale) Mirror 22: 0x4a6df067 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc714: skipped 000495 1788838885.520085 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc714): Mirror 11: 0x4a6df067 (stale) Mirror 12: 0x4a6df067 (stale) Mirror 13: 0x4a6df067 (stale) Mirror 16: 0x4a6df067 (stale) Mirror 17: 0x4a6df067 (stale) Mirror 18: 0x4a6df067 (stale) Mirror 19: 0x4a6df067 (stale) Mirror 20: 0x4a6df067 (stale) Mirror 22: 0x4a6df067 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc714): Mirror 11: 0x4a6df067 (stale) Mirror 12: 0x4a6df067 (stale) Mirror 13: 0x4a6df067 (stale) Mirror 16: 0x4a6df067 (stale) Mirror 17: 0x4a6df067 (stale) Mirror 18: 0x4a6df067 (stale) Mirror 19: 0x4a6df067 (stale) Mirror 20: 0x4a6df067 (stale) Mirror 22: 0x4a6df067 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc714: skipped 000498 1788838886.464102 punch from 0x0048e4 to 0x00c713 (0x7e30 bytes) 000499 1788838886.471996 punch from 0x005da2 to 0x00a78e (0x49ed bytes) 000500 1788838886.481230 read 0x0012bd thru 0x00138d (0xd1 bytes) 000501 1788838886.490185 write 0x0155af thru 0x01942a (0x3e7c bytes) 000502 1788838886.492193 mapwrite 0x037aab thru 0x03c67b (0x4bd1 bytes) 000504 1788838886.635966 write 0x0365ce thru 0x03ffff (0x9a32 bytes) 000505 1788838886.641836 trunc from 0x040000 to 0x02b405 (0x14bfa bytes) 000506 1788838886.693219 mapwrite 0x011cb8 thru 0x020d87 (0xf0d0 bytes) 000508 1788838886.852422 mapwrite 0x019346 thru 0x01f245 (0x5f00 bytes) 000511 1788838886.928578 write 0x031e22 thru 0x036ca2 (0x4e81 bytes) 000512 1788838886.934142 write 0x00c16d thru 0x00ccd2 (0xb66 bytes) 000513 1788838886.936297 write 0x01d636 thru 0x02056f (0x2f3a bytes) 000514 1788838886.937935 mapread 0x00887b thru 0x014872 (0xbff8 bytes) 000515 1788838886.950004 trunc from 0x036ca3 to 0x01a1ba (0x1cae8 bytes) 000516 1788838886.975036 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a1bb): Mirror 11: 0x9b6e59d5 (stale) Mirror 12: 0x9b6e59d5 (stale) Mirror 13: 0x9b6e59d5 (stale) Mirror 16: 0x9b6e59d5 (stale) Mirror 17: 0x9b6e59d5 (stale) Mirror 19: 0x9b6e59d5 (stale) Mirror 20: 0x9b6e59d5 (stale) Mirror 22: 0x9b6e59d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a1bb): Mirror 11: 0x9b6e59d5 (stale) Mirror 12: 0x9b6e59d5 (stale) Mirror 13: 0x9b6e59d5 (stale) Mirror 16: 0x9b6e59d5 (stale) Mirror 17: 0x9b6e59d5 (stale) Mirror 19: 0x9b6e59d5 (stale) Mirror 20: 0x9b6e59d5 (stale) Mirror 22: 0x9b6e59d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a1bb: skipped 000517 1788838887.570767 read 0x0034f8 thru 0x00b793 (0x829c bytes) 000518 1788838887.594475 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a1bb): Mirror 12: 0x9b6e59d5 (stale) Mirror 13: 0x9b6e59d5 (stale) Mirror 16: 0x9b6e59d5 (stale) Mirror 17: 0x9b6e59d5 (stale) Mirror 19: 0x9b6e59d5 (stale) Mirror 20: 0x9b6e59d5 (stale) Mirror 22: 0x9b6e59d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a1bb): Mirror 12: 0x9b6e59d5 (stale) Mirror 13: 0x9b6e59d5 (stale) Mirror 16: 0x9b6e59d5 (stale) Mirror 17: 0x9b6e59d5 (stale) Mirror 19: 0x9b6e59d5 (stale) Mirror 20: 0x9b6e59d5 (stale) Mirror 22: 0x9b6e59d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a1bb: skipped 000520 1788838888.305893 mapwrite 0x025710 thru 0x03454e (0xee3f bytes) 000521 1788838888.466351 write 0x02e382 thru 0x02ecad (0x92c bytes) 000522 1788838888.468372 mapwrite 0x00f853 thru 0x010e82 (0x1630 bytes) 000523 1788838888.525904 punch from 0x01573c to 0x01ecea (0x95af bytes) 000524 1788838888.532123 read 0x0225c0 thru 0x0272a3 (0x4ce4 bytes) 000525 1788838888.538992 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3454f): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3454f): Mirror 12: 0xea8e2967 (stale) Mirror 13: 0xea8e2967 (stale) Mirror 16: 0xea8e2967 (stale) Mirror 17: 0xea8e2967 (stale) Mirror 19: 0xea8e2967 (stale) Mirror 20: 0xea8e2967 (stale) Mirror 22: 0xea8e2967 (stale) CRC-32 checksum value for chunk [0, 0x3454f): Mirror 1: 0x1f067930 Mirror 23: 0x1f067930 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3454f: skipped 000526 1788838890.000200 write 0x028f7b thru 0x036a6a (0xdaf0 bytes) 000527 1788838890.152297 mapwrite 0x02c04d thru 0x03bd2b (0xfcdf bytes) 000528 1788838890.310842 mapwrite 0x01e0ec thru 0x021eed (0x3e02 bytes) 000529 1788838890.409730 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd2c): Mirror 12: 0x20b41537 (stale) Mirror 13: 0x20b41537 (stale) Mirror 16: 0x20b41537 (stale) Mirror 17: 0x20b41537 (stale) Mirror 20: 0x20b41537 (stale) Mirror 22: 0x20b41537 (stale) Mirror 23: 0x343bae4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd2c): Mirror 12: 0x20b41537 (stale) Mirror 13: 0x20b41537 (stale) Mirror 16: 0x20b41537 (stale) Mirror 17: 0x20b41537 (stale) Mirror 20: 0x20b41537 (stale) Mirror 22: 0x20b41537 (stale) Mirror 23: 0x343bae4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd2c: skipped 000530 1788838891.230911 mapread 0x00a8ad thru 0x016b85 (0xc2d9 bytes) 000531 1788838891.273467 read 0x01bb0d thru 0x026799 (0xac8d bytes) 000533 1788838891.287304 trunc from 0x03bd2c to 0x03357a (0x87b1 bytes) 000534 1788838891.315161 read 0x0208f4 thru 0x027e04 (0x7511 bytes) 000535 1788838891.338065 punch from 0x016cf5 to 0x01a4d0 (0x37dc bytes) 000536 1788838891.345566 mapread 0x028536 thru 0x03036f (0x7e3a bytes) 000537 1788838891.357028 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3357b): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3357b): Mirror 12: 0x40e1b095 (stale) Mirror 13: 0x40e1b095 (stale) Mirror 16: 0x40e1b095 (stale) Mirror 17: 0x40e1b095 (stale) Mirror 20: 0x40e1b095 (stale) Mirror 22: 0x40e1b095 (stale) Mirror 23: 0x10c0defe (stale) CRC-32 checksum value for chunk [0, 0x3357b): Mirror 1: 0xda4d0cfc Mirror 24: 0xda4d0cfc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3357b: skipped 000540 1788838892.863995 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3357b): Mirror 12: 0x40e1b095 (stale) Mirror 13: 0x40e1b095 (stale) Mirror 16: 0x40e1b095 (stale) Mirror 17: 0x40e1b095 (stale) Mirror 20: 0x40e1b095 (stale) Mirror 23: 0x10c0defe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3357b): Mirror 12: 0x40e1b095 (stale) Mirror 13: 0x40e1b095 (stale) Mirror 16: 0x40e1b095 (stale) Mirror 17: 0x40e1b095 (stale) Mirror 20: 0x40e1b095 (stale) Mirror 23: 0x10c0defe (stale) CRC-32 checksum value for chunk [0, 0x3357b): Mirror 1: 0xda4d0cfc Mirror 24: 0xda4d0cfc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3357b: skipped 000541 1788838894.451887 read 0x007c0d thru 0x00ea53 (0x6e47 bytes) 000543 1788838894.492453 punch from 0x028eb6 to 0x02b66a (0x27b5 bytes) 000544 1788838894.612331 trunc from 0x03357b to 0x035917 (0x239d bytes) 000545 1788838894.664025 mapread 0x02b148 thru 0x035917 (0xa7d0 bytes) 000546 1788838894.686946 mapread 0x01e918 thru 0x02b96a (0xd053 bytes) 000547 1788838894.697791 trunc from 0x035918 to 0x027aa3 (0xde74 bytes) 000548 1788838894.720890 mapwrite 0x03854c thru 0x03ffff (0x7ab4 bytes) 000549 1788838894.809211 write 0x0340eb thru 0x0396f6 (0x560c bytes) 000550 1788838894.810549 read 0x01ae20 thru 0x01c0a2 (0x1283 bytes) 000551 1788838894.817079 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa6118cd5 (stale) Mirror 13: 0xa6118cd5 (stale) Mirror 16: 0xa6118cd5 (stale) Mirror 17: 0xa6118cd5 (stale) Mirror 20: 0xa6118cd5 (stale) Mirror 24: 0x1c545c53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xa6118cd5 (stale) Mirror 13: 0xa6118cd5 (stale) Mirror 16: 0xa6118cd5 (stale) Mirror 17: 0xa6118cd5 (stale) Mirror 20: 0xa6118cd5 (stale) Mirror 24: 0x1c545c53 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000553 1788838895.356068 mapread 0x012c52 thru 0x01bbce (0x8f7d bytes) 000554 1788838895.381063 trunc from 0x040000 to 0x010944 (0x2f6bb bytes) 000555 1788838895.401762 punch from 0x00b3fd to 0x010944 (0x5548 bytes) 000558 1788838895.416559 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10945): Mirror 13: 0xa1632d33 (stale) Mirror 16: 0xa1632d33 (stale) Mirror 17: 0xa1632d33 (stale) Mirror 20: 0xa1632d33 (stale) Mirror 24: 0xb6f04ff8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10945): Mirror 13: 0xa1632d33 (stale) Mirror 16: 0xa1632d33 (stale) Mirror 17: 0xa1632d33 (stale) Mirror 20: 0xa1632d33 (stale) Mirror 24: 0xb6f04ff8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10945: skipped 000559 1788838895.922606 write 0x001f6d thru 0x00412c (0x21c0 bytes) 000560 1788838895.937172 mapwrite 0x006312 thru 0x015cde (0xf9cd bytes) 000561 1788838896.074251 mapread 0x0097c3 thru 0x012354 (0x8b92 bytes) 000563 1788838896.085981 read 0x002621 thru 0x004564 (0x1f44 bytes) 000567 1788838896.101314 trunc from 0x015cdf to 0x00c8b9 (0x9425 bytes) 000568 1788838896.135722 mapwrite 0x00d95e thru 0x011112 (0x37b5 bytes) 000569 1788838896.238749 write 0x01bb1f thru 0x028f97 (0xd479 bytes) 000571 1788838896.244917 punch from 0x0005d4 to 0x00bb93 (0xb5c0 bytes) 000572 1788838896.279080 read 0x0133c9 thru 0x018ae3 (0x571b bytes) 000573 1788838896.298829 read 0x01c9de thru 0x028f97 (0xc5ba bytes) 000574 1788838896.300070 mapread 0x005ca9 thru 0x00abd7 (0x4f2f bytes) 000576 1788838896.302100 write 0x00ec1e thru 0x013c5f (0x5042 bytes) 000577 1788838896.303431 punch from 0x005ef7 to 0x014574 (0xe67e bytes) 000579 1788838896.332836 mapwrite 0x0116d4 thru 0x0184bf (0x6dec bytes) 000581 1788838896.461015 trunc from 0x028f98 to 0x0149ff (0x14598 bytes) 000583 1788838896.519645 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14a00): Mirror 16: 0x2a02fca5 (stale) Mirror 17: 0x2a02fca5 (stale) Mirror 20: 0x2a02fca5 (stale) Mirror 24: 0x3c532bfa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14a00): Mirror 16: 0x2a02fca5 (stale) Mirror 17: 0x2a02fca5 (stale) Mirror 20: 0x2a02fca5 (stale) Mirror 24: 0x3c532bfa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14a00: skipped 000584 1788838897.247606 mapwrite 0x00ed92 thru 0x00f522 (0x791 bytes) 000586 1788838897.341731 MIRROR_RESYNC 5 mirrors 000587 1788838899.001033 MIRROR_EXTEND to 6 mirrors 000588 1788838899.811622 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000589 1788838900.421751 mapwrite 0x00e1ab thru 0x01973a (0xb590 bytes) 000591 1788838900.595845 write 0x022c91 thru 0x02bdaf (0x911f bytes) 000592 1788838900.602809 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bdb0): Mirror 16: 0x88da3ba9 (stale) Mirror 17: 0x88da3ba9 (stale) Mirror 24: 0x88da3ba9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bdb0): Mirror 16: 0x88da3ba9 (stale) Mirror 17: 0x88da3ba9 (stale) Mirror 24: 0x88da3ba9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bdb0: skipped 000593 1788838901.032374 write 0x03bf05 thru 0x03ffff (0x40fb bytes) 000596 1788838901.039370 read 0x03a489 thru 0x03ffff (0x5b77 bytes) 000597 1788838901.049473 read 0x02c482 thru 0x02ff00 (0x3a7f bytes) 000599 1788838901.083728 mapwrite 0x031c3c thru 0x03868a (0x6a4f bytes) 000601 1788838901.193613 MIRROR_RESYNC 4 mirrors 000602 1788838902.554039 mapread 0x019b51 thru 0x028fe5 (0xf495 bytes) 000604 1788838902.587048 MIRROR_RESYNC 4 mirrors 000607 1788838903.202926 trunc from 0x040000 to 0x03dc81 (0x237e bytes) 000609 1788838903.361836 read 0x000080 thru 0x006220 (0x61a1 bytes) 000610 1788838903.389092 punch from 0x02bf8d to 0x03b738 (0xf7ac bytes) 000613 1788838903.404301 punch from 0x00b10f to 0x010e29 (0x5d1b bytes) 000614 1788838903.430607 trunc from 0x03dc82 to 0x01bf07 (0x21d7a bytes) 000615 1788838903.487475 mapwrite 0x00a240 thru 0x00ddf3 (0x3bb4 bytes) 000616 1788838903.668698 MIRROR_RESYNC 4 mirrors 000617 1788838905.146860 write 0x020b2d thru 0x02efd3 (0xe4a7 bytes) 000618 1788838905.244370 trunc from 0x02efd4 to 0x011517 (0x1dabc bytes) 000619 1788838905.280208 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11518): Mirror 17: 0x8bf2ed3a (stale) Mirror 24: 0x8bf2ed3a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11518): Mirror 17: 0x8bf2ed3a (stale) Mirror 24: 0x8bf2ed3a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11518: skipped 000620 1788838905.826797 mapread 0x005f1b thru 0x008082 (0x2168 bytes) 000622 1788838905.859773 mapread 0x00a1cb thru 0x011517 (0x734d bytes) 000623 1788838905.867923 punch from 0x004754 to 0x010752 (0xbfff bytes) 000624 1788838905.887189 trunc from 0x011518 to 0x01c33c (0xae25 bytes) 000625 1788838905.931564 mapwrite 0x0006ae thru 0x007033 (0x6986 bytes) 000626 1788838906.066118 read 0x014014 thru 0x01c33c (0x8329 bytes) 000627 1788838906.094997 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c33d): Mirror 24: 0x33e44636 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c33d): Mirror 24: 0x33e44636 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c33d: skipped 000628 1788838906.511267 punch from 0x010f9c to 0x01bf56 (0xafbb bytes) 000629 1788838906.518712 mapread 0x006346 thru 0x0122c5 (0xbf80 bytes) 000630 1788838906.553642 read 0x016aea thru 0x01aa5b (0x3f72 bytes) 000631 1788838906.559328 punch from 0x01207b to 0x016e83 (0x4e09 bytes) 000632 1788838906.566297 write 0x029cbf thru 0x034bf0 (0xaf32 bytes) 000633 1788838906.568055 read 0x0038cf thru 0x004ed9 (0x160b bytes) 000634 1788838906.573551 MIRROR_RESYNC 2 mirrors 000635 1788838907.513997 mapread 0x02cbe3 thru 0x034bf0 (0x800e bytes) 000636 1788838907.559140 MIRROR_EXTEND to 3 mirrors 000638 1788838908.448804 trunc from 0x034bf1 to 0x011140 (0x23ab0 bytes) 000639 1788838908.606269 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11141): Mirror 25: 0x29378417 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11141): Mirror 25: 0x29378417 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11141: skipped 000640 1788838909.049083 mapread 0x00bfb2 thru 0x011140 (0x518f bytes) 000641 1788838909.082493 MIRROR_RESYNC 2 mirrors 000643 1788838910.037648 read 0x003d20 thru 0x009cdf (0x5fc0 bytes) 000648 1788838910.055682 mapwrite 0x003ad2 thru 0x013843 (0xfd72 bytes) 000649 1788838910.375039 trunc from 0x013844 to 0x037de6 (0x245a3 bytes) 000650 1788838910.416777 mapwrite 0x009d4a thru 0x01137b (0x7632 bytes) 000651 1788838910.499623 write 0x005638 thru 0x00711b (0x1ae4 bytes) 000652 1788838910.501312 trunc from 0x037de7 to 0x036066 (0x1d80 bytes) 000653 1788838910.547477 read 0x0310f3 thru 0x036066 (0x4f74 bytes) 000655 1788838910.576773 write 0x02a364 thru 0x031292 (0x6f2f bytes) 000656 1788838910.578205 read 0x01f0b2 thru 0x029039 (0x9f88 bytes) 000657 1788838910.583907 mapread 0x00dd59 thru 0x019608 (0xb8b0 bytes) 000666 1788838910.611698 write 0x0264cb thru 0x027498 (0xfce bytes) 000668 1788838910.621128 punch from 0x005bbc to 0x011020 (0xb465 bytes) 000669 1788838910.659661 write 0x0064e9 thru 0x00d153 (0x6c6b bytes) 000670 1788838910.691102 punch from 0x00fafc to 0x01e507 (0xea0c bytes) 000671 1788838910.730712 mapwrite 0x03c54a thru 0x03ffff (0x3ab6 bytes) 000672 1788838910.938719 trunc from 0x040000 to 0x02e09e (0x11f61 bytes) 000673 1788838910.981126 MIRROR_RESYNC 2 mirrors 000677 1788838912.192996 trunc from 0x02e09f to 0x02300b (0xb093 bytes) 000678 1788838912.381134 write 0x0178f4 thru 0x02341c (0xbb29 bytes) 000680 1788838912.415530 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2341d): Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2341d): Mirror 25: 0x619ebaa3 (stale) CRC-32 checksum value for chunk [0, 0x2341d): Mirror 1: 0x75b1f6b7 Mirror 26: 0x75b1f6b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2341d: skipped 000682 1788838914.027429 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2341d): Mirror 25: 0x619ebaa3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2341d): Mirror 25: 0x619ebaa3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2341d: skipped 000683 1788838914.323814 trunc from 0x02341d to 0x0305b9 (0xd19d bytes) 000684 1788838914.355341 mapread 0x0240d6 thru 0x02e21e (0xa149 bytes) 000685 1788838914.378761 mapread 0x0139c9 thru 0x0193bf (0x59f7 bytes) 000687 1788838914.389620 write 0x016081 thru 0x024dce (0xed4e bytes) 000688 1788838914.392056 read 0x007739 thru 0x016939 (0xf201 bytes) 000689 1788838914.411995 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x305ba): Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x305ba): Mirror 25: 0x138ebce9 (stale) CRC-32 checksum value for chunk [0, 0x305ba): Mirror 1: 0x7701b925 Mirror 26: 0x7701b925 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x305ba: skipped 000690 1788838915.968557 mapwrite 0x0376e7 thru 0x03ffff (0x8919 bytes) 000693 1788838916.228005 trunc from 0x040000 to 0x00db46 (0x324b9 bytes) 000694 1788838916.265598 mapwrite 0x00ef0b thru 0x016e28 (0x7f1e bytes) 000695 1788838916.401074 write 0x00e269 thru 0x01dd25 (0xfabd bytes) 000696 1788838916.402790 read 0x00d3a8 thru 0x0136b7 (0x6310 bytes) 000697 1788838916.414461 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1dd26): Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dd26): Mirror 25: 0x928eb43a (stale) Mirror 26: 0xe353822e (stale) CRC-32 checksum value for chunk [0, 0x1dd26): Mirror 1: 0x16cd5124 Mirror 27: 0x16cd5124 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dd26: skipped 000700 1788838917.523532 read 0x003ff9 thru 0x00a6a9 (0x66b1 bytes) 000701 1788838917.550826 write 0x03aee4 thru 0x03d086 (0x21a3 bytes) 000702 1788838917.662428 mapread 0x005d2b thru 0x00a7cc (0x4aa2 bytes) 000704 1788838917.719303 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d087): Mirror 26: 0x12492abe (stale) Mirror 27: 0xb10ca612 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d087): Mirror 26: 0x12492abe (stale) Mirror 27: 0xb10ca612 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d087: skipped 000706 1788838918.325784 punch from 0x01ad3c to 0x026cbe (0xbf83 bytes) 000707 1788838918.332480 mapwrite 0x00efab thru 0x01bf42 (0xcf98 bytes) 000708 1788838918.449547 read 0x02e6e5 thru 0x03d086 (0xe9a2 bytes) 000710 1788838918.459517 punch from 0x02a37e to 0x032d58 (0x89db bytes) 000711 1788838918.469566 punch from 0x039879 to 0x039bf8 (0x380 bytes) 000712 1788838918.474736 trunc from 0x03d087 to 0x0106e6 (0x2c9a0 bytes) 000713 1788838918.521269 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x106e7): Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x106e7): Mirror 26: 0x4218107f (stale) Mirror 27: 0x5f8d2edf (stale) CRC-32 checksum value for chunk [0, 0x106e7): Mirror 1: 0x89d7e0fc Mirror 28: 0x89d7e0fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x106e7: skipped 000714 1788838919.617316 read 0x007b1a thru 0x0106e6 (0x8bcd bytes) 000715 1788838919.636588 trunc from 0x0106e7 to 0x005f68 (0xa77e bytes) 000716 1788838919.759693 read 0x0009bf thru 0x005f68 (0x55aa bytes) 000718 1788838919.779829 write 0x0361e3 thru 0x03c6f2 (0x6510 bytes) 000719 1788838919.788788 punch from 0x02c0e2 to 0x030bee (0x4b0d bytes) 000721 1788838919.808050 trunc from 0x03c6f3 to 0x022107 (0x1a5eb bytes) 000723 1788838919.842974 read 0x018863 thru 0x01bfbe (0x375c bytes) 000725 1788838919.869150 write 0x009daa thru 0x00c95f (0x2bb6 bytes) 000726 1788838919.871953 punch from 0x013289 to 0x01ce9c (0x9c14 bytes) 000728 1788838919.886198 mapread 0x0187ac thru 0x022107 (0x995c bytes) 000730 1788838919.945014 read 0x015357 thru 0x01ddfe (0x8aa8 bytes) 000731 1788838919.961985 mapwrite 0x036118 thru 0x03ce97 (0x6d80 bytes) 000732 1788838920.098481 mapread 0x00d02f thru 0x012c5e (0x5c30 bytes) 000733 1788838920.099489 mapwrite 0x004d13 thru 0x0052c1 (0x5af bytes) 000734 1788838920.165833 read 0x00a340 thru 0x00ed00 (0x49c1 bytes) 000735 1788838920.167770 punch from 0x030776 to 0x03a6d3 (0x9f5e bytes) 000736 1788838920.178690 read 0x021193 thru 0x02f0bc (0xdf2a bytes) 000737 1788838920.179540 punch from 0x0393cd to 0x03ce97 (0x3acb bytes) 000738 1788838920.196782 MIRROR_RESYNC 4 mirrors 000739 1788838921.590939 write 0x00f756 thru 0x010041 (0x8ec bytes) 000740 1788838921.665938 trunc from 0x03ce98 to 0x00ea52 (0x2e445 bytes) 000741 1788838921.706548 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xea53): Mirror 27: 0xd8ccdb91 (stale) Mirror 28: 0xd8ccdb91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xea53): Mirror 27: 0xd8ccdb91 (stale) Mirror 28: 0xd8ccdb91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xea53: skipped 000742 1788838922.084227 write 0x03c351 thru 0x03ffff (0x3caf bytes) 000743 1788838922.103745 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 28: 0xe9db62a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 28: 0xe9db62a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000744 1788838922.611258 trunc from 0x040000 to 0x00d9b5 (0x3264a bytes) 000746 1788838922.640323 punch from 0x000084 to 0x0026ba (0x2637 bytes) 000747 1788838922.648265 read 0x00494f thru 0x00d9b5 (0x9067 bytes) 000748 1788838922.665618 write 0x028857 thru 0x028d6c (0x516 bytes) 000749 1788838922.668115 trunc from 0x028d6d to 0x022d84 (0x5fe8 bytes) 000750 1788838922.706336 mapwrite 0x019fad thru 0x0282eb (0xe33f bytes) 000751 1788838922.948551 mapwrite 0x03186d thru 0x03ffff (0xe793 bytes) 000752 1788838923.140410 write 0x0354c5 thru 0x03b6b4 (0x61f0 bytes) 000753 1788838923.162155 MIRROR_RESYNC 2 mirrors 000754 1788838924.101855 mapread 0x03e60f thru 0x03ffff (0x19f1 bytes) 000756 1788838924.121515 read 0x00adbb thru 0x014520 (0x9766 bytes) 000759 1788838924.148089 read 0x0303ae thru 0x039c24 (0x9877 bytes) 000761 1788838924.168433 mapread 0x03b00a thru 0x03ce91 (0x1e88 bytes) 000763 1788838924.169488 trunc from 0x040000 to 0x03de14 (0x21eb bytes) 000764 1788838924.378513 trunc from 0x03de15 to 0x01dead (0x1ff67 bytes) 000765 1788838924.453042 mapwrite 0x037f96 thru 0x03ffff (0x806a bytes) 000768 1788838924.653457 mapread 0x03d333 thru 0x03ffff (0x2ccd bytes) 000770 1788838924.657759 write 0x02c804 thru 0x02d2a2 (0xa9f bytes) 000771 1788838924.668895 mapwrite 0x0333b2 thru 0x039611 (0x6260 bytes) 000773 1788838924.751076 trunc from 0x040000 to 0x00f208 (0x30df7 bytes) 000774 1788838924.769995 mapwrite 0x00c4fd thru 0x011d5d (0x5861 bytes) 000777 1788838924.896851 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x11d5e): Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d5e): Mirror 28: 0x9700a1e3 (stale) CRC-32 checksum value for chunk [0, 0x11d5e): Mirror 1: 0x5d6fa46f Mirror 29: 0x5d6fa46f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d5e: skipped 000778 1788838926.566111 mapread 0x0000a5 thru 0x001d81 (0x1cdd bytes) 000782 1788838926.580922 read 0x005e03 thru 0x011d5d (0xbf5b bytes) 000784 1788838926.589570 punch from 0x0024f6 to 0x00c2cd (0x9dd8 bytes) 000786 1788838926.713873 punch from 0x00e3d0 to 0x011d5d (0x398e bytes) 000787 1788838926.726995 trunc from 0x011d5e to 0x02c31b (0x1a5be bytes) 000788 1788838926.777565 trunc from 0x02c31c to 0x00f1ed (0x1d12e bytes) 000789 1788838926.830217 mapread 0x00bd0a thru 0x00e214 (0x250b bytes) 000790 1788838926.888715 write 0x013ebd thru 0x015fd5 (0x2119 bytes) 000791 1788838926.889983 read 0x015858 thru 0x015fd5 (0x77e bytes) 000792 1788838926.903765 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15fd6): Mirror 28: 0x50a3f050 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15fd6): Mirror 28: 0x50a3f050 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15fd6: skipped 000793 1788838927.428041 trunc from 0x015fd6 to 0x0365b6 (0x205e1 bytes) 000796 1788838927.474067 read 0x024478 thru 0x02501c (0xba5 bytes) 000797 1788838927.482060 punch from 0x01943b to 0x02543a (0xc000 bytes) 000798 1788838927.492122 trunc from 0x0365b7 to 0x0371b9 (0xc03 bytes) 000800 1788838927.537769 mapwrite 0x0077ea thru 0x010401 (0x8c18 bytes) 000802 1788838927.695164 write 0x023cfe thru 0x028038 (0x433b bytes) 000803 1788838927.712553 write 0x011186 thru 0x01e2f6 (0xd171 bytes) 000806 1788838927.740133 mapwrite 0x013ea8 thru 0x023dcb (0xff24 bytes) 000807 1788838927.850648 mapwrite 0x001053 thru 0x002f77 (0x1f25 bytes) 000808 1788838927.930439 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000809 1788838928.073805 trunc from 0x0371ba to 0x00b852 (0x2b967 bytes) 000811 1788838928.098138 mapread 0x008777 thru 0x00b852 (0x30dc bytes) 000812 1788838928.125780 punch from 0x00ab8d to 0x00b852 (0xcc6 bytes) 000813 1788838928.132525 punch from 0x0035ea to 0x00b852 (0x8269 bytes) 000814 1788838928.141993 mapwrite 0x02dfbf thru 0x037736 (0x9778 bytes) 000818 1788838928.269145 write 0x03e626 thru 0x03ffff (0x19da bytes) 000819 1788838928.272178 mapread 0x0196ac thru 0x026088 (0xc9dd bytes) 000821 1788838928.320815 read 0x035c20 thru 0x03ffff (0xa3e0 bytes) 000822 1788838928.332573 mapread 0x01f131 thru 0x02ecf5 (0xfbc5 bytes) 000824 1788838928.340514 read 0x03f743 thru 0x03ffff (0x8bd bytes) 000826 1788838928.348343 trunc from 0x040000 to 0x01d346 (0x22cb9 bytes) 000830 1788838928.395997 mapread 0x011955 thru 0x01d346 (0xb9f2 bytes) 000833 1788838928.430619 punch from 0x012a5f to 0x01918d (0x672f bytes) 000836 1788838928.442949 trunc from 0x01d347 to 0x01441e (0x8f28 bytes) 000837 1788838928.494261 write 0x02ad42 thru 0x02cab7 (0x1d76 bytes) 000838 1788838928.495487 punch from 0x0091a8 to 0x012754 (0x95ad bytes) 000842 1788838928.519561 mapwrite 0x00e858 thru 0x01e0d4 (0xf87d bytes) 000844 1788838928.844072 trunc from 0x02cab8 to 0x0397c5 (0xcd0e bytes) 000846 1788838928.893968 mapread 0x012797 thru 0x01c361 (0x9bcb bytes) 000847 1788838928.903317 mapread 0x03148e thru 0x0397c5 (0x8338 bytes) 000848 1788838928.916934 write 0x0186a1 thru 0x01ca1c (0x437c bytes) 000849 1788838928.923859 mapwrite 0x027861 thru 0x036ccf (0xf46f bytes) 000851 1788838929.067133 read 0x00cc0e thru 0x014335 (0x7728 bytes) 000852 1788838929.095000 trunc from 0x0397c6 to 0x0119e8 (0x27ddd bytes) 000854 1788838929.137847 punch from 0x00c4e3 to 0x00fe95 (0x39b3 bytes) 000856 1788838929.156010 write 0x027690 thru 0x02f4c3 (0x7e34 bytes) 000857 1788838929.160510 mapwrite 0x00e4cd thru 0x0124c2 (0x3ff6 bytes) 000858 1788838929.351728 read 0x01b884 thru 0x0263a3 (0xab20 bytes) 000859 1788838929.382868 trunc from 0x02f4c4 to 0x016ecf (0x185f4 bytes) 000860 1788838929.441950 read 0x00923b thru 0x013be0 (0xa9a6 bytes) 000863 1788838929.446456 trunc from 0x016ed0 to 0x01f0c7 (0x81f8 bytes) 000866 1788838929.493036 mapwrite 0x02d7e8 thru 0x0325bc (0x4dd5 bytes) 000868 1788838929.666001 mapread 0x017da8 thru 0x01c406 (0x465f bytes) 000870 1788838929.670742 mapread 0x01633c thru 0x018faa (0x2c6f bytes) 000872 1788838929.674603 trunc from 0x0325bd to 0x0179c7 (0x1abf5 bytes) 000874 1788838929.711129 mapread 0x01053d thru 0x0179c7 (0x748b bytes) 000875 1788838929.713553 write 0x031b4f thru 0x03ffff (0xe4b1 bytes) 000876 1788838929.715577 punch from 0x02388a to 0x033283 (0xf9fa bytes) 000877 1788838929.750921 punch from 0x03a0d6 to 0x03ffff (0x5f2a bytes) 000879 1788838929.771999 mapwrite 0x006bb6 thru 0x01285e (0xbca9 bytes) 000880 1788838930.027896 mapread 0x0385ce thru 0x0390c5 (0xaf8 bytes) 000883 1788838930.049976 trunc from 0x040000 to 0x03526e (0xad91 bytes) 000884 1788838930.120939 read 0x00a78b thru 0x015826 (0xb09c bytes) 000887 1788838930.128033 mapwrite 0x016e10 thru 0x024076 (0xd267 bytes) 000888 1788838930.262788 write 0x0134fd thru 0x01d79a (0xa29e bytes) 000889 1788838930.271193 MIRROR_EXTEND to 2 mirrors 000890 1788838931.213642 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) 000891 1788838931.396168 mapread 0x00e1d4 thru 0x01645b (0x8288 bytes) 000892 1788838931.418917 read 0x01ba7a thru 0x021fe2 (0x6569 bytes) 000893 1788838931.426987 read 0x0049ee thru 0x00d41e (0x8a31 bytes) 000894 1788838931.434345 read 0x00dd33 thru 0x018bb5 (0xae83 bytes) 000896 1788838931.435605 mapread 0x029e1c thru 0x02d3e3 (0x35c8 bytes) 000897 1788838931.437604 mapwrite 0x03dce2 thru 0x03ffff (0x231e bytes) 000898 1788838931.604721 MIRROR_EXTEND to 2 mirrors 000899 1788838932.571047 read 0x0375d4 thru 0x03ffff (0x8a2c bytes) 000901 1788838932.605206 write 0x030a83 thru 0x03516c (0x46ea bytes) 000904 1788838932.707192 mapread 0x036408 thru 0x03ffff (0x9bf8 bytes) 000905 1788838932.736965 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x927797fc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x90f3fb40 Mirror 3: 0x90f3fb40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000907 1788838934.304297 trunc from 0x040000 to 0x005edb (0x3a124 bytes) 000909 1788838934.440880 read 0x0043a3 thru 0x005edb (0x1b39 bytes) 000914 1788838934.451198 trunc from 0x005edc to 0x033e4a (0x2df6f bytes) 000915 1788838934.471553 mapread 0x02b835 thru 0x033e4a (0x8616 bytes) 000917 1788838934.495870 mapread 0x0171bb thru 0x024ad4 (0xd91a bytes) 000918 1788838934.521022 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33e4b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e4b): Mirror 2: 0x40a467db (stale) Mirror 3: 0x293b650 (stale) CRC-32 checksum value for chunk [0, 0x33e4b): Mirror 1: 0x863894b3 Mirror 4: 0x863894b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e4b: skipped 000919 1788838936.060619 trunc from 0x033e4b to 0x016e15 (0x1d035 bytes) 000920 1788838936.212363 trunc from 0x016e16 to 0x0072a8 (0xfb6d bytes) 000921 1788838936.238881 mapwrite 0x02ea40 thru 0x03551f (0x6ae0 bytes) 000922 1788838936.368821 mapread 0x02987f thru 0x031cc8 (0x844a bytes) 000925 1788838936.372688 read 0x02eb42 thru 0x03551f (0x69de bytes) 000928 1788838936.375671 punch from 0x002914 to 0x004c8d (0x237a bytes) 000929 1788838936.381591 read 0x02f411 thru 0x03237e (0x2f6e bytes) 000933 1788838936.385836 write 0x028ba5 thru 0x02f985 (0x6de1 bytes) 000934 1788838936.401074 punch from 0x0318b2 to 0x03551f (0x3c6e bytes) 000935 1788838936.417952 trunc from 0x035520 to 0x012c3c (0x228e3 bytes) 000938 1788838936.451124 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x12c3d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12c3d): Mirror 2: 0xe318829c (stale) Mirror 3: 0xe318829c (stale) Mirror 4: 0xd3bccfc8 (stale) CRC-32 checksum value for chunk [0, 0x12c3d): Mirror 1: 0x45841e4e Mirror 5: 0x45841e4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12c3d: skipped 000939 1788838938.367245 read 0x0046b4 thru 0x00e8cf (0xa21c bytes) 000940 1788838938.426138 punch from 0x0062dd to 0x01184f (0xb573 bytes) 000941 1788838938.650685 trunc from 0x012c3d to 0x01eb4a (0xbf0e bytes) 000942 1788838938.707169 mapwrite 0x01096a thru 0x01397d (0x3014 bytes) 000944 1788838938.946898 mapread 0x004af8 thru 0x010493 (0xb99c bytes) 000945 1788838938.967659 read 0x003d53 thru 0x00fb93 (0xbe41 bytes) 000946 1788838938.970066 write 0x02ecfb thru 0x037125 (0x842b bytes) 000949 1788838938.980057 mapread 0x007441 thru 0x00b70b (0x42cb bytes) 000950 1788838938.984088 mapread 0x00d9d0 thru 0x00e61d (0xc4e bytes) 000952 1788838938.990980 mapwrite 0x025d0d thru 0x03575b (0xfa4f bytes) 000953 1788838939.178368 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37126): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37126): Mirror 2: 0xc2a01cd7 (stale) Mirror 3: 0x231deb15 (stale) Mirror 4: 0x2a6d85be (stale) Mirror 5: 0x2fc843f8 (stale) CRC-32 checksum value for chunk [0, 0x37126): Mirror 1: 0x540bd83a Mirror 6: 0x540bd83a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37126: skipped 000954 1788838941.005329 mapread 0x03597b thru 0x037125 (0x17ab bytes) 000956 1788838941.104729 read 0x0051b6 thru 0x00bc29 (0x6a74 bytes) 000957 1788838941.162804 trunc from 0x037126 to 0x0282c3 (0xee62 bytes) 000960 1788838941.285356 mapread 0x00d20b thru 0x019e4a (0xcc40 bytes) 000961 1788838941.321726 mapread 0x007505 thru 0x008e93 (0x198f bytes) 000962 1788838941.325995 write 0x03caa2 thru 0x03ffff (0x355e bytes) 000963 1788838941.327680 write 0x025405 thru 0x02b92f (0x652b bytes) 000964 1788838941.344314 punch from 0x007092 to 0x011660 (0xa5cf bytes) 000965 1788838941.375017 mapread 0x02c822 thru 0x0390f9 (0xc8d8 bytes) 000966 1788838941.412060 read 0x01c211 thru 0x022918 (0x6708 bytes) 000968 1788838941.432490 mapwrite 0x005f56 thru 0x0065d4 (0x67f bytes) 000970 1788838941.484845 punch from 0x036758 to 0x03a88b (0x4134 bytes) 000972 1788838941.493495 mapread 0x0244bf thru 0x026c22 (0x2764 bytes) 000973 1788838941.494949 punch from 0x018b33 to 0x024c63 (0xc131 bytes) 000974 1788838941.519491 trunc from 0x040000 to 0x0386ca (0x7935 bytes) 000975 1788838941.571898 write 0x02b1b4 thru 0x037397 (0xc1e4 bytes) 000976 1788838941.588584 mapread 0x012c45 thru 0x0143bc (0x1778 bytes) 000977 1788838941.591590 read 0x01cb75 thru 0x028505 (0xb991 bytes) 000978 1788838941.613330 mapread 0x032d0a thru 0x0386ca (0x59c1 bytes) 000979 1788838941.614993 punch from 0x00e3b3 to 0x018223 (0x9e71 bytes) 000980 1788838941.658946 write 0x0317bb thru 0x035318 (0x3b5e bytes) 000981 1788838941.663459 read 0x0316ad thru 0x0386ca (0x701e bytes) 000982 1788838941.664436 mapwrite 0x02d884 thru 0x02f280 (0x19fd bytes) 000983 1788838941.755252 read 0x022883 thru 0x0278ea (0x5068 bytes) 000984 1788838941.756212 mapwrite 0x017022 thru 0x01f158 (0x8137 bytes) 000985 1788838941.926896 read 0x006114 thru 0x015d93 (0xfc80 bytes) 000986 1788838941.944442 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x386cb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x386cb): Mirror 2: 0xb98442c0 (stale) Mirror 3: 0xd2ece05a (stale) Mirror 4: 0xde6875a9 (stale) Mirror 5: 0x66dbf83c (stale) Mirror 6: 0xfe570328 (stale) CRC-32 checksum value for chunk [0, 0x386cb): Mirror 1: 0x3386c83b Mirror 7: 0x3386c83b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x386cb: skipped 000987 1788838943.689582 punch from 0x023ba6 to 0x02dc43 (0xa09e bytes) 000989 1788838943.778267 mapread 0x031292 thru 0x0386ca (0x7439 bytes) 000992 1788838943.799737 trunc from 0x0386cb to 0x00a12d (0x2e59d bytes) 000993 1788838943.820676 mapread 0x0029d4 thru 0x00a12d (0x775a bytes) 000994 1788838943.860370 punch from 0x000e74 to 0x002fd5 (0x2162 bytes) 000995 1788838943.874582 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x186:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa12e): Mirror 2: 0x45dac194 (stale) Mirror 3: 0x45dac194 (stale) Mirror 4: 0xd2655375 (stale) Mirror 5: 0xa1242dab (stale) Mirror 6: 0xa1242dab (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa12e): Mirror 2: 0x45dac194 (stale) Mirror 3: 0x45dac194 (stale) Mirror 4: 0xd2655375 (stale) Mirror 5: 0xa1242dab (stale) Mirror 6: 0xa1242dab (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa12e: skipped 000996 1788838944.627693 punch from 0x002be6 to 0x00a12d (0x7548 bytes) 000997 1788838944.639916 write 0x00c593 thru 0x015942 (0x93b0 bytes) 000998 1788838944.642312 trunc from 0x015943 to 0x003c1c (0x11d26 bytes) 000999 1788838944.674749 read 0x0007e4 thru 0x0024c8 (0x1ce5 bytes) 001000 1788838944.688429 write 0x00c059 thru 0x01092f (0x48d7 bytes) All operations completed A-OK! PASS 70a (152s) == sanity-flr test 71: check mirror extend parent fid ==== 23:42:30 (1788838950) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 23:42:34 (1788838954) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.340232 s, 6.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304168 s, 3.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 23:42:44 (1788838964) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre2 Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3578560 .. Extending file size to 5149120 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5813184 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6290144 .. Extending file size to 6418432 .. Extending file size to 6549952 .. Extending file size to 8662048 .. Extending file size to 7289216 .. Extending file size to 8677504 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7964192 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8343456 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8750240 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8715808 .. Extending file size to 8996096 .. Waiting 57432 57434 57435 57436 57437 failed 192.168.204.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre2 (opts:) 192.168.204.121@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:44:06 (1788839046) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre2 Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5781184 .. Extending file size to 3592928 .. done Extending file size to 4766816 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6274272 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7047616 .. Extending file size to 7504000 .. Extending file size to 8452320 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9073888 .. Extending file size to 7879552 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8139744 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59029 59030 59032 59034 59035 done 192.168.204.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre2 (opts:) 192.168.204.121@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (90s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:45:36 (1788839136) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101407 s, 20.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.343635 s, 6.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x2e0131cd (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:45:56 (1788839156) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0999264 s, 21.0 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 23:46:04 (1788839164) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:46:13 (1788839173) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228518 s, 9.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:46:21 (1788839181) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269553 s, 7.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:46:28 (1788839188) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205672 s, 10.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:47:04 (1788839224) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg421-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199847 s, 5.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } PASS 204d (67s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:48:11 (1788839291) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg421-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259539 s, 8.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (61s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:49:13 (1788839353) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg421-server mds-ost sync done. Creating new pool test_204f oleg421-server: Pool lustre.test_204f created Adding targets to pool oleg421-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg421-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.502915 s, 4.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg421-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg421-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg421-server: Pool lustre.test_204f destroyed PASS 204f (88s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:50:41 (1788839441) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:50:50 (1788839450) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:50:58 (1788839458) oleg421-server: Pool lustre.test_206 created Adding targets to pool oleg421-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg421-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg421-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg421-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg421-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:51:24 (1788839484) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844448 s, 12.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608584 s, 17.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4dd9a8ca Mirror 3: 0x4dd9a8ca PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:51:37 (1788839497) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:51:42 (1788839502) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:51:47 (1788839507) Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0873262 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1990000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2080000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2080000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2080000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x20a0000 Read this (4 bytes): BBBB 192.168.204.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg421-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:52:30 (1788839550) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0291602 s, 281 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:52:39 (1788839559) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703984 s, 14.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x203 lmm_fid: [0x200000404:0x203:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x203 lmm_fid: [0x200000404:0x203:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834435 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:52:41.000000000 -0400 Modify: 2026-09-07 23:52:41.000000000 -0400 Change: 2026-09-07 23:52:43.000000000 -0400 Birth: 2026-09-07 23:52:42.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:52:50 (1788839570) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:52:55 (1788839575) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.654018 s, 62.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.152738 s, 26.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 23:52 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x205:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 23:52 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:53:05 (1788839585) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:53:16 (1788839596) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 1956 sec =========== 23:53:28 (1788839608) === sanity-flr: start cleanup 23:53:30 (1788839610) === === sanity-flr: finish cleanup 23:53:38 (1788839618) ===