-----============= acceptance-small: sanity-flr ============----- Thu Oct 1 18:59:25 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg438-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 18:59:46 (1790895586) === oleg438-client.virtnet: executing check_config_client /mnt/lustre oleg438-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg438-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c9244e5c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c9244e5c800.idle_timeout=debug disable quota as required oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:00:32 (1790895632) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7627] == sanity-flr test 0a: lfs mirror create with -N option == 19:00:34 (1790895634) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:00:52 (1790895652) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:00:57 (1790895657) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (5s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:01:02 (1790895662) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:01:30 (1790895690) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:01:33 (1790895693) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 19:01:37 (1790895697) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c9244e5c800.stats=clear osc.lustre-OST0001-osc-ffff8c9244e5c800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:01:47 (1790895707) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:01:58 (1790895718) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:02:09 (1790895729) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:02:15 (1790895735) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:02:25 (1790895745) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 19:02:36 (1790895756) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:02:44 (1790895764) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:02:49 (1790895769) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:02:59 (1790895779) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:03:12 (1790895792) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:03:20 (1790895800) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.90199 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239599 s, 4.4 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:03:35 (1790895815) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.214075 s, 29.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:03:44 (1790895824) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:03:56 (1790895836) Stopping /mnt/lustre-ost1 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c9244e5c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c9244e5c800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg438-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg438-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c9244e5c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c9244e5c800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg438-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (136s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:06:11 (1790895971) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.221769 s, 18.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c9244e5c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c9244e5c800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg438-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:06:55 (1790896015) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c9244e5c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c9244e5c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg438-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c9244e5c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c9244e5c800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg438-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:08:16 (1790896096) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.77027 s, 13.9 MB/s -rw-r--r-- 1 root root 52428800 Oct 1 19:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.190s user 0m0.001s sys 0m0.167s Stopping /mnt/lustre-ost1 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c9244e5c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c9244e5c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.841s user 0m0.001s sys 0m0.172s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg438-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg438-server oleg438-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c9244e5c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c9244e5c800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.502s user 0m0.000s sys 0m0.205s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg438-server: oleg438-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg438-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg438-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c9244e5c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c9244e5c800.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:09:52 (1790896192) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:09:55 (1790896195) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:09:59 (1790896199) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 19:10:02 (1790896202) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 19:10:10 (1790896210) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.535871 s, 23.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:10:23 (1790896223) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605059 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034882 s, 30.1 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:10:32 (1790896232) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:10:41 (1790896241) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0014439 s, 709 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00262581 s, 390 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00207493 s, 494 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00135939 s, 753 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 19:10:51 (1790896251) create test files with size 15739251 .. 9d42c73b4c7fd06f21a121ead6a470e5 - 5897035ece01dda86844043f3a8f7993 - 0f5b85d21447a74e67e93181d27bead2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg438-client.virtnet /mnt/lustre (opts:) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 19:11:13 (1790896273) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:11:30 (1790896290) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:11:37 (1790896297) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139141 s, 7.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696451 s, 15.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc3:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc4:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 19:11:47 (1790896307) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242791 s, 8.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.543449 s, 7.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.340382 s, 6.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 19:12:07 (1790896327) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 19:12:11 (1790896331) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:12:14 (1790896334) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433592 s, 24.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 19:12:21 (1790896341) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:12:25 (1790896345) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe2:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:12:36 (1790896356) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.295814 s, 35.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg438-client.virtnet /mnt/lustre (opts:) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:12:47 (1790896367) llite.lustre-ffff8c9243fe2000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.280742 s, 37.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg438-client.virtnet /mnt/lustre (opts:) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8c9250921800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 19:13:02 (1790896382) error lfs split: create victim file failed: File exists PASS 44f (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:13:22 (1790896402) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:14:00 (1790896440) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:14:44 (1790896484) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:14:47 (1790896487) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.303973 s, 10.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.259582 s, 12.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7b6e51a4 (nosync,stale) Mirror 2: 0x7b6e51a4 (nosync,stale) ** verify mirror 2 doesn't change original checksum: a07c2fa7371024e7ef8c985cf74e5749 - mirror 1 checksum: b37ad0615b0dd3fdb8a2e1e193267a6c - mirror 2 checksum: a07c2fa7371024e7ef8c985cf74e5749 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-01T19:14:51-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x10a:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-01T19:14:51-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x10b:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a07c2fa7371024e7ef8c985cf74e5749 - mirror 1 checksum: b37ad0615b0dd3fdb8a2e1e193267a6c - mirror 2 checksum: b37ad0615b0dd3fdb8a2e1e193267a6c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-01T19:14:51-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x10a:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-01T19:14:51-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x10b:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:15:01 (1790896501) Failing mds1 on oleg438-server Stopping /mnt/lustre-mds1 (opts:) on oleg438-server 19:15:09 (1790896509) shut down facet: mds1 facet_host: oleg438-server facet_failover_host: oleg438-server Failover mds1 to oleg438-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg438-server: oleg438-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg438-client: oleg438-server: ssh exited with exit code 1 Started lustre-MDT0000 19:15:40 (1790896540) targets are mounted 19:15:40 (1790896540) facet_failover done oleg438-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037382 s, 28.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x69 lmm_fid: [0x200000404:0x69:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x10c:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:16:05 (1790896565) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:16:08 (1790896568) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:16:12 (1790896572) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0264707 s, 619 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:16:20 (1790896580) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:16:24 (1790896584) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.024136 s, 170 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0826093 s, 49.6 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:16:34 (1790896594) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:16:59 (1790896619) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:17:29 (1790896649) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa7:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:17:53 (1790896673) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:18:17 (1790896697) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 19:18:24 (1790896704) mirror create pid 55031 mirror split pid 55032 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 55031 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 55032 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 19:19:32 (1790896772) Seed set to 6774 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1790896774.366360 mapwrite 0x02e4e1 thru 0x0338ab (0x53cb bytes) 000004 1790896774.454380 punch from 0x00a18e to 0x016bfd (0xca70 bytes) truncating to largest ever: 0xb462 000006 1790896774.460096 trunc from 0x0338ac to 0x00b461 (0x2844a bytes) 000008 1790896774.483573 mapread 0x009d51 thru 0x00b461 (0x1711 bytes) 000010 1790896774.492588 mapread 0x004801 thru 0x006f14 (0x2714 bytes) 000011 1790896774.493609 read 0x003c40 thru 0x006b4d (0x2f0e bytes) truncating to largest ever: 0xbc51 000012 1790896774.495151 trunc from 0x00b462 to 0x00bc50 (0x7ef bytes) 000013 1790896774.511188 mapread 0x00a04a thru 0x00bc50 (0x1c07 bytes) 000015 1790896774.513641 read 0x002b7a thru 0x005928 (0x2daf bytes) 000017 1790896774.515322 mapread 0x0084ec thru 0x00bc50 (0x3765 bytes) 000018 1790896774.517133 trunc from 0x00bc51 to 0x008afe (0x3152 bytes) 000020 1790896774.535688 read 0x004e1a thru 0x008afe (0x3ce5 bytes) 000021 1790896774.536247 write 0x01f715 thru 0x027715 (0x8001 bytes) 000023 1790896774.539122 mapread 0x000ac1 thru 0x00d6fd (0xcc3d bytes) 000024 1790896774.546946 write 0x018e89 thru 0x019fb0 (0x1128 bytes) 000025 1790896774.547971 write 0x034da2 thru 0x03ffff (0xb25e bytes) truncating to largest ever: 0x3c757 000026 1790896774.550525 trunc from 0x040000 to 0x03c756 (0x38a9 bytes) 000027 1790896774.570371 punch from 0x02e836 to 0x02f74a (0xf15 bytes) 000028 1790896774.597239 mapread 0x0216ab thru 0x02db52 (0xc4a8 bytes) 000029 1790896774.635692 trunc from 0x03c757 to 0x01c5bc (0x2019a bytes) 000031 1790896774.654188 mapread 0x001bff thru 0x00b2a3 (0x96a5 bytes) 000032 1790896774.658376 punch from 0x00a5ff to 0x00aa87 (0x489 bytes) 000034 1790896774.684653 MIRROR_EXTEND to 2 mirrors 000035 1790896775.498655 trunc from 0x01c5bd to 0x03bc82 (0x1f6c6 bytes) 000036 1790896775.581007 trunc from 0x03bc83 to 0x037c89 (0x3ff9 bytes) 000037 1790896775.607991 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37c8a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37c8a): Mirror 2: 0xf9943878 (stale) CRC-32 checksum value for chunk [0, 0x37c8a): Mirror 1: 0xf9943878 Mirror 3: 0xf9943878 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37c8a: skipped 000038 1790896776.768623 MIRROR_RESYNC 3 mirrors 000039 1790896777.659298 read 0x00bed2 thru 0x00f4a0 (0x35cf bytes) 000040 1790896777.672687 read 0x036bc3 thru 0x037c89 (0x10c7 bytes) 000041 1790896777.681564 mapread 0x000121 thru 0x00afc1 (0xaea1 bytes) 000045 1790896777.688947 MIRROR_EXTEND to 4 mirrors 000048 1790896778.513759 mapwrite 0x034bda thru 0x03ffff (0xb426 bytes) 000049 1790896778.804565 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaffd5457 (stale) Mirror 4: 0xaffd5457 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaffd5457 (stale) Mirror 4: 0xaffd5457 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000050 1790896779.252114 mapread 0x0298cc thru 0x0375b2 (0xdce7 bytes) 000052 1790896779.298494 mapwrite 0x02d6f9 thru 0x03c183 (0xea8b bytes) 000054 1790896779.360884 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaffd5457 (stale) Mirror 4: 0xaffd5457 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc8e5d831 Mirror 5: 0xc8e5d831 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000055 1790896780.545724 read 0x027461 thru 0x0361e6 (0xed86 bytes) 000057 1790896780.580779 mapwrite 0x01cbae thru 0x01ffd0 (0x3423 bytes) 000058 1790896780.793043 trunc from 0x040000 to 0x027bf1 (0x1840e bytes) 000059 1790896780.829824 write 0x014ef2 thru 0x016904 (0x1a13 bytes) 000061 1790896780.841458 write 0x0391e6 thru 0x03965e (0x479 bytes) 000067 1790896780.853604 mapread 0x000525 thru 0x002de0 (0x28bc bytes) 000068 1790896780.867591 trunc from 0x03965f to 0x004729 (0x34f35 bytes) 000070 1790896780.904491 punch from 0x0006d4 to 0x004729 (0x4056 bytes) 000073 1790896780.919525 write 0x034555 thru 0x03ffff (0xbaab bytes) 000074 1790896780.924550 read 0x039807 thru 0x03ffff (0x67f9 bytes) 000075 1790896780.932231 mapwrite 0x018efb thru 0x021ca7 (0x8dad bytes) 000080 1790896780.994726 trunc from 0x040000 to 0x0368fa (0x9705 bytes) 000083 1790896781.018322 punch from 0x02468a to 0x02882d (0x41a4 bytes) 000084 1790896781.023525 trunc from 0x0368fb to 0x0125a2 (0x24358 bytes) 000085 1790896781.042959 trunc from 0x0125a3 to 0x0364c5 (0x23f23 bytes) 000086 1790896781.067881 mapwrite 0x01ecc6 thru 0x021ff5 (0x3330 bytes) truncating to largest ever: 0x3cb21 000088 1790896781.133644 trunc from 0x0364c6 to 0x03cb20 (0x665b bytes) 000091 1790896781.158026 mapwrite 0x0244a2 thru 0x02f49a (0xaff9 bytes) 000092 1790896781.302910 read 0x02a010 thru 0x02b7a5 (0x1796 bytes) 000093 1790896781.303693 punch from 0x01ee8b to 0x02681b (0x7991 bytes) 000094 1790896781.307927 read 0x015fe7 thru 0x01cd75 (0x6d8f bytes) 000095 1790896781.323057 write 0x014fee thru 0x015b34 (0xb47 bytes) 000096 1790896781.324266 read 0x03c6eb thru 0x03cb20 (0x436 bytes) 000098 1790896781.329308 mapwrite 0x0146b9 thru 0x018d4e (0x4696 bytes) 000099 1790896781.406228 write 0x024711 thru 0x02db5d (0x944d bytes) 000100 1790896781.408175 punch from 0x02f2af to 0x03cb20 (0xd872 bytes) 000101 1790896781.418905 trunc from 0x03cb21 to 0x00d26a (0x2f8b6 bytes) 000102 1790896781.448415 mapwrite 0x00a0c2 thru 0x0167ad (0xc6ec bytes) 000103 1790896781.593860 mapwrite 0x02ba93 thru 0x032627 (0x6b95 bytes) 000104 1790896781.801357 mapwrite 0x02bef4 thru 0x039bf7 (0xdd04 bytes) 000106 1790896781.941748 mapread 0x0040c0 thru 0x00bc1b (0x7b5c bytes) 000107 1790896781.947642 punch from 0x016871 to 0x017309 (0xa99 bytes) 000108 1790896781.955341 read 0x027f62 thru 0x029123 (0x11c2 bytes) 000109 1790896781.960612 MIRROR_RESYNC 4 mirrors 000110 1790896783.453709 punch from 0x00f88c to 0x01181b (0x1f90 bytes) truncating to largest ever: 0x3fd83 000111 1790896783.556996 trunc from 0x039bf8 to 0x03fd82 (0x618b bytes) 000112 1790896783.581287 punch from 0x039898 to 0x03fd82 (0x64eb bytes) 000113 1790896783.587695 mapread 0x01f0da thru 0x024a1f (0x5946 bytes) 000114 1790896783.612084 mapread 0x0328eb thru 0x03bafb (0x9211 bytes) 000116 1790896783.628771 mapwrite 0x02261d thru 0x028f29 (0x690d bytes) 000118 1790896783.694309 punch from 0x027d09 to 0x02b542 (0x383a bytes) 000120 1790896783.705357 read 0x021a77 thru 0x0261be (0x4748 bytes) 000121 1790896783.706184 punch from 0x018896 to 0x024247 (0xb9b2 bytes) 000122 1790896783.711126 punch from 0x000578 to 0x004814 (0x429d bytes) 000124 1790896783.722542 read 0x02db61 thru 0x03a5be (0xca5e bytes) 000126 1790896783.744390 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fd83): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd83): Mirror 3: 0x6c9e8546 (stale) Mirror 4: 0x6c9e8546 (stale) Mirror 5: 0x6c9e8546 (stale) CRC-32 checksum value for chunk [0, 0x3fd83): Mirror 1: 0x6f6f56c3 Mirror 6: 0x6f6f56c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd83: skipped 000127 1790896785.171936 punch from 0x002187 to 0x00c445 (0xa2bf bytes) 000128 1790896785.345548 trunc from 0x03fd83 to 0x018c3c (0x27146 bytes) 000130 1790896785.382297 mapread 0x002d08 thru 0x0108ea (0xdbe3 bytes) 000131 1790896785.401694 write 0x039dbb thru 0x03ffff (0x6245 bytes) 000133 1790896785.403819 write 0x00e8ba thru 0x019a3c (0xb183 bytes) 000134 1790896785.410910 read 0x00a5ce thru 0x00e49a (0x3ecd bytes) 000135 1790896785.412109 mapwrite 0x003982 thru 0x004365 (0x9e4 bytes) 000136 1790896785.503372 mapread 0x0031f2 thru 0x004652 (0x1461 bytes) 000138 1790896785.504758 punch from 0x01a2b4 to 0x0268c7 (0xc614 bytes) 000139 1790896785.510431 mapwrite 0x01e6f0 thru 0x027fae (0x98bf bytes) 000140 1790896785.579535 read 0x036392 thru 0x03ffff (0x9c6e bytes) 000141 1790896785.590288 mapwrite 0x030f83 thru 0x034c79 (0x3cf7 bytes) 000143 1790896785.667393 mapwrite 0x024d6c thru 0x02871d (0x39b2 bytes) 000144 1790896785.721160 write 0x03448d thru 0x03666e (0x21e2 bytes) 000145 1790896785.722623 mapwrite 0x016125 thru 0x0162aa (0x186 bytes) 000146 1790896785.778404 read 0x01e61f thru 0x02160b (0x2fed bytes) 000147 1790896785.779707 mapwrite 0x019b4f thru 0x01e29e (0x4750 bytes) 000148 1790896785.815038 trunc from 0x040000 to 0x039f9c (0x6063 bytes) 000149 1790896785.835100 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39f9d): Mirror 4: 0xec72e3f0 (stale) Mirror 5: 0xec72e3f0 (stale) Mirror 6: 0x20844a91 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39f9d): Mirror 4: 0xec72e3f0 (stale) Mirror 5: 0xec72e3f0 (stale) Mirror 6: 0x20844a91 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39f9d: skipped 000150 1790896786.175048 read 0x0053d8 thru 0x00edd4 (0x99fd bytes) 000153 1790896786.184711 read 0x020757 thru 0x02fb6a (0xf414 bytes) 000154 1790896786.209508 punch from 0x036fd5 to 0x039f9c (0x2fc8 bytes) 000156 1790896786.217112 trunc from 0x039f9d to 0x01a2b7 (0x1fce5 bytes) 000157 1790896786.256654 MIRROR_RESYNC 4 mirrors 000159 1790896787.158948 punch from 0x00b4f4 to 0x00d4ae (0x1fbb bytes) 000160 1790896787.223234 write 0x025c71 thru 0x028747 (0x2ad7 bytes) 000166 1790896787.236910 trunc from 0x028748 to 0x02700e (0x1739 bytes) 000167 1790896787.284733 mapwrite 0x0214eb thru 0x026997 (0x54ad bytes) 000168 1790896787.387843 read 0x019bd1 thru 0x01a623 (0xa53 bytes) 000170 1790896787.411495 trunc from 0x02700f to 0x03a0c5 (0x130b7 bytes) 000171 1790896787.433784 MIRROR_RESYNC 4 mirrors 000172 1790896788.492820 mapread 0x025668 thru 0x02b2f8 (0x5c91 bytes) 000173 1790896788.514623 MIRROR_RESYNC 4 mirrors 000174 1790896788.962572 trunc from 0x03a0c6 to 0x011c26 (0x2849f bytes) 000177 1790896789.061223 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11c27): Mirror 5: 0xeef745c6 (stale) Mirror 6: 0xeef745c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11c27): Mirror 5: 0xeef745c6 (stale) Mirror 6: 0xeef745c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11c27: skipped 000179 1790896789.416544 read 0x00925b thru 0x00f0de (0x5e84 bytes) 000180 1790896789.425475 read 0x000c3d thru 0x00aa5d (0x9e21 bytes) 000182 1790896789.437823 read 0x004bb1 thru 0x011c26 (0xd076 bytes) 000185 1790896789.440312 mapread 0x008cff thru 0x00ef51 (0x6253 bytes) 000186 1790896789.441426 mapwrite 0x0335b9 thru 0x03b89f (0x82e7 bytes) 000187 1790896789.519902 mapwrite 0x024ee1 thru 0x02c49e (0x75be bytes) 000188 1790896789.588540 mapwrite 0x038065 thru 0x03ffff (0x7f9b bytes) 000189 1790896789.663345 mapread 0x018d71 thru 0x01f116 (0x63a6 bytes) 000190 1790896789.668744 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8b985ce3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8b985ce3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000192 1790896790.035238 mapwrite 0x01d388 thru 0x028126 (0xad9f bytes) 000195 1790896790.121525 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8b985ce3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc37c7fda Mirror 7: 0xc37c7fda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000196 1790896790.836842 punch from 0x02ded5 to 0x032773 (0x489f bytes) 000198 1790896790.922721 write 0x03bc2e thru 0x03ffff (0x43d2 bytes) 000199 1790896790.930349 write 0x013e2c thru 0x023b66 (0xfd3b bytes) 000200 1790896790.947429 write 0x010ca1 thru 0x01c60d (0xb96d bytes) 000201 1790896790.975490 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8b985ce3 (stale) Mirror 7: 0xc37c7fda (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9a229f5e Mirror 8: 0x9a229f5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000202 1790896791.982326 read 0x017aec thru 0x01b36a (0x387f bytes) 000203 1790896792.007254 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8b985ce3 (stale) Mirror 7: 0xc37c7fda (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9a229f5e Mirror 8: 0x9a229f5e Mirror 9: 0x9a229f5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000204 1790896792.892440 write 0x01ecbd thru 0x0217da (0x2b1e bytes) 000206 1790896792.985733 write 0x03ceeb thru 0x03ffff (0x3115 bytes) 000207 1790896793.007120 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8b985ce3 (stale) Mirror 8: 0x9a229f5e (stale) Mirror 9: 0x9a229f5e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8b985ce3 (stale) Mirror 8: 0x9a229f5e (stale) Mirror 9: 0x9a229f5e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000208 1790896793.314186 punch from 0x01be59 to 0x022521 (0x66c9 bytes) 000209 1790896793.318292 punch from 0x010a35 to 0x0135da (0x2ba6 bytes) 000210 1790896793.324645 read 0x0277e7 thru 0x031f28 (0xa742 bytes) 000211 1790896793.334280 mapwrite 0x008cef thru 0x00937f (0x691 bytes) 000212 1790896793.452343 mapwrite 0x03faad thru 0x03ffff (0x553 bytes) 000213 1790896793.499877 mapread 0x0281f9 thru 0x02e9bb (0x67c3 bytes) 000214 1790896793.501251 trunc from 0x040000 to 0x021af8 (0x1e507 bytes) 000216 1790896793.518989 trunc from 0x021af9 to 0x03b2d5 (0x197dd bytes) 000218 1790896793.542552 write 0x011399 thru 0x01189a (0x502 bytes) 000219 1790896793.555282 read 0x02733d thru 0x02872c (0x13f0 bytes) 000222 1790896793.570061 read 0x0006e1 thru 0x00e2aa (0xdbca bytes) 000223 1790896793.570708 read 0x01567e thru 0x0168a5 (0x1228 bytes) 000224 1790896793.575147 MIRROR_RESYNC 4 mirrors 000225 1790896794.598182 read 0x001616 thru 0x004040 (0x2a2b bytes) 000230 1790896794.604463 trunc from 0x03b2d6 to 0x03f3cb (0x40f6 bytes) 000231 1790896794.690536 mapwrite 0x02e133 thru 0x02e7ad (0x67b bytes) 000232 1790896794.732328 mapread 0x01ce0f thru 0x02b9f7 (0xebe9 bytes) 000235 1790896794.763849 mapread 0x0213a8 thru 0x027b39 (0x6792 bytes) 000238 1790896794.766190 punch from 0x000919 to 0x00af78 (0xa660 bytes) 000239 1790896794.772150 trunc from 0x03f3cc to 0x02c2c5 (0x13106 bytes) 000241 1790896794.798587 read 0x00539b thru 0x00ccdb (0x7941 bytes) 000242 1790896794.821041 read 0x0025d9 thru 0x006dea (0x4812 bytes) 000244 1790896794.822388 punch from 0x021e34 to 0x022b8e (0xd5b bytes) 000245 1790896794.830884 mapread 0x019f3d thru 0x02408c (0xa150 bytes) 000246 1790896794.842040 read 0x01e95d thru 0x025d63 (0x7407 bytes) 000247 1790896794.842645 mapwrite 0x00fd26 thru 0x01ec7d (0xef58 bytes) 000248 1790896794.928890 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c2c6): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c2c6): Mirror 6: 0x23b6f963 (stale) Mirror 8: 0x23b6f963 (stale) Mirror 9: 0x23b6f963 (stale) CRC-32 checksum value for chunk [0, 0x2c2c6): Mirror 1: 0x55299a6 Mirror 10: 0x55299a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c2c6: skipped 000249 1790896795.728007 MIRROR_RESYNC 5 mirrors 000251 1790896796.663097 write 0x037320 thru 0x03f9a7 (0x8688 bytes) 000255 1790896796.750842 trunc from 0x03f9a8 to 0x002732 (0x3d275 bytes) 000256 1790896796.787052 read 0x002027 thru 0x002732 (0x70c bytes) 000258 1790896796.795944 mapread 0x001a36 thru 0x002732 (0xcfd bytes) 000260 1790896796.810786 mapread 0x002714 thru 0x002732 (0x1f bytes) 000261 1790896796.811645 write 0x022258 thru 0x02e626 (0xc3cf bytes) 000263 1790896796.817890 write 0x026429 thru 0x0288e6 (0x24be bytes) 000264 1790896796.820196 mapwrite 0x0194fb thru 0x01e487 (0x4f8d bytes) 000265 1790896796.931164 read 0x025996 thru 0x027087 (0x16f2 bytes) 000266 1790896796.931833 trunc from 0x02e627 to 0x019a44 (0x14be2 bytes) 000268 1790896796.962841 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x19a45): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19a45): Mirror 6: 0x442ee4e2 (stale) Mirror 8: 0x442ee4e2 (stale) Mirror 9: 0x442ee4e2 (stale) Mirror 10: 0x442ee4e2 (stale) CRC-32 checksum value for chunk [0, 0x19a45): Mirror 1: 0xdfda9e1b Mirror 11: 0xdfda9e1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19a45: skipped 000269 1790896798.194951 MIRROR_RESYNC 6 mirrors 000271 1790896799.533449 punch from 0x015b1e to 0x0195fb (0x3ade bytes) 000272 1790896799.607078 write 0x00a3ea thru 0x00c0ea (0x1d01 bytes) 000273 1790896799.620048 punch from 0x019960 to 0x019a44 (0xe5 bytes) 000274 1790896799.627916 mapwrite 0x0154aa thru 0x01f127 (0x9c7e bytes) 000276 1790896799.732624 write 0x008e88 thru 0x016aeb (0xdc64 bytes) 000277 1790896799.743521 mapread 0x002acf thru 0x00b958 (0x8e8a bytes) 000278 1790896799.756100 trunc from 0x01f128 to 0x01c28f (0x2e98 bytes) 000280 1790896799.772700 trunc from 0x01c290 to 0x03bca5 (0x1fa16 bytes) 000281 1790896799.796421 MIRROR_RESYNC 6 mirrors 000283 1790896801.567010 write 0x00508b thru 0x00ee7a (0x9df0 bytes) 000284 1790896801.648643 punch from 0x03486d to 0x03bca5 (0x7439 bytes) 000285 1790896801.663076 mapwrite 0x007c40 thru 0x00b6bc (0x3a7d bytes) 000286 1790896801.707649 mapread 0x021eac thru 0x024096 (0x21eb bytes) 000289 1790896801.729296 read 0x00e222 thru 0x0170c6 (0x8ea5 bytes) 000290 1790896801.738450 mapread 0x00a967 thru 0x010932 (0x5fcc bytes) 000294 1790896801.742142 trunc from 0x03bca6 to 0x01f149 (0x1cb5c bytes) 000295 1790896801.768218 mapwrite 0x004e5c thru 0x00851d (0x36c2 bytes) 000297 1790896801.824267 mapwrite 0x03297d thru 0x03cf24 (0xa5a8 bytes) 000298 1790896801.901612 punch from 0x01377d to 0x01a3ff (0x6c83 bytes) 000299 1790896801.907753 trunc from 0x03cf25 to 0x031c34 (0xb2f0 bytes) 000300 1790896801.932085 MIRROR_RESYNC 6 mirrors 000301 1790896803.630133 write 0x026d0b thru 0x03019f (0x9495 bytes) 000303 1790896803.730317 mapread 0x01e424 thru 0x024703 (0x62e0 bytes) 000304 1790896803.754564 read 0x0299fe thru 0x031c34 (0x8237 bytes) 000306 1790896803.768536 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31c35): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c35): Mirror 6: 0x311633df (stale) Mirror 8: 0x311633df (stale) Mirror 9: 0x311633df (stale) Mirror 10: 0x311633df (stale) Mirror 11: 0x311633df (stale) CRC-32 checksum value for chunk [0, 0x31c35): Mirror 1: 0xceb354f5 Mirror 12: 0xceb354f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c35: skipped 000308 1790896804.759479 mapwrite 0x001397 thru 0x009e65 (0x8acf bytes) 000309 1790896805.001358 mapwrite 0x0387a0 thru 0x03ffff (0x7860 bytes) 000310 1790896805.081097 trunc from 0x040000 to 0x02673c (0x198c3 bytes) 000312 1790896805.097104 read 0x01d555 thru 0x02540e (0x7eba bytes) 000314 1790896805.107448 punch from 0x011cc7 to 0x017a75 (0x5daf bytes) 000317 1790896805.115003 mapread 0x01436a thru 0x01911a (0x4db1 bytes) 000318 1790896805.125652 read 0x01ee90 thru 0x02673c (0x78ad bytes) 000320 1790896805.133602 trunc from 0x02673d to 0x02b58e (0x4e52 bytes) 000322 1790896805.156168 write 0x027998 thru 0x035a82 (0xe0eb bytes) 000323 1790896805.170902 MIRROR_RESYNC 7 mirrors 000324 1790896806.406324 mapwrite 0x002bab thru 0x00b304 (0x875a bytes) 000327 1790896806.569507 punch from 0x026a3a to 0x034076 (0xd63d bytes) 000328 1790896806.574565 punch from 0x001454 to 0x00c156 (0xad03 bytes) 000329 1790896806.582010 trunc from 0x035a83 to 0x023379 (0x12709 bytes) 000331 1790896806.652517 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2337a): Mirror 8: 0x7fb4dcd6 (stale) Mirror 9: 0x7fb4dcd6 (stale) Mirror 10: 0x7fb4dcd6 (stale) Mirror 11: 0x7fb4dcd6 (stale) Mirror 12: 0x7fb4dcd6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2337a): Mirror 8: 0x7fb4dcd6 (stale) Mirror 9: 0x7fb4dcd6 (stale) Mirror 10: 0x7fb4dcd6 (stale) Mirror 11: 0x7fb4dcd6 (stale) Mirror 12: 0x7fb4dcd6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2337a: skipped 000332 1790896807.165509 mapwrite 0x02bf72 thru 0x034a31 (0x8ac0 bytes) 000333 1790896807.344536 MIRROR_RESYNC 6 mirrors 000334 1790896808.816763 mapread 0x00ecdd thru 0x015bce (0x6ef2 bytes) 000335 1790896808.841988 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) 000336 1790896809.269931 mapread 0x02b1c2 thru 0x0310d8 (0x5f17 bytes) 000337 1790896809.293347 mapwrite 0x0194a4 thru 0x01edde (0x593b bytes) 000340 1790896809.489155 read 0x0044d1 thru 0x00b460 (0x6f90 bytes) 000341 1790896809.501528 mapwrite 0x0349ca thru 0x036847 (0x1e7e bytes) 000342 1790896809.576408 punch from 0x00d013 to 0x0154bf (0x84ad bytes) 000344 1790896809.581348 write 0x037669 thru 0x03c5e1 (0x4f79 bytes) 000345 1790896809.582887 trunc from 0x03c5e2 to 0x03989f (0x2d42 bytes) 000346 1790896809.607313 mapread 0x016edd thru 0x018b00 (0x1c24 bytes) 000349 1790896809.612769 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x398a0): Mirror 9: 0xa22a9a55 (stale) Mirror 11: 0xa22a9a55 (stale) Mirror 12: 0xa22a9a55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398a0): Mirror 9: 0xa22a9a55 (stale) Mirror 11: 0xa22a9a55 (stale) Mirror 12: 0xa22a9a55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398a0: skipped 000352 1790896810.047372 write 0x03e8b4 thru 0x03ffff (0x174c bytes) 000353 1790896810.048469 write 0x0237f5 thru 0x025a8c (0x2298 bytes) 000354 1790896810.062050 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x441ef6c9 (stale) Mirror 11: 0x441ef6c9 (stale) Mirror 12: 0x441ef6c9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xff432f9d Mirror 13: 0xff432f9d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000357 1790896811.010572 read 0x03855c thru 0x03a415 (0x1eba bytes) 000360 1790896811.044367 read 0x03a7cf thru 0x03ffff (0x5831 bytes) 000361 1790896811.060442 punch from 0x02b8b7 to 0x03516d (0x98b7 bytes) 000362 1790896811.172334 write 0x03bf66 thru 0x03ffff (0x409a bytes) 000363 1790896811.179648 trunc from 0x040000 to 0x01cc87 (0x23378 bytes) 000364 1790896811.211958 read 0x012250 thru 0x012ebc (0xc6d bytes) 000365 1790896811.268016 mapread 0x00cd62 thru 0x01ab9f (0xde3e bytes) 000369 1790896811.270985 mapread 0x00d26a thru 0x015aa1 (0x8838 bytes) 000370 1790896811.280380 write 0x004ae7 thru 0x012b5b (0xe075 bytes) 000371 1790896811.286336 mapread 0x00e3fd thru 0x0139c1 (0x55c5 bytes) 000372 1790896811.289287 trunc from 0x01cc88 to 0x03aecc (0x1e245 bytes) 000373 1790896811.313524 read 0x027981 thru 0x033b24 (0xc1a4 bytes) 000375 1790896811.341264 mapwrite 0x018433 thru 0x02446d (0xc03b bytes) 000376 1790896811.473393 mapread 0x004310 thru 0x010765 (0xc456 bytes) 000378 1790896811.481928 read 0x039a2f thru 0x03aecc (0x149e bytes) 000379 1790896811.482715 punch from 0x010f92 to 0x013e7c (0x2eeb bytes) 000380 1790896811.488573 mapwrite 0x00e51d thru 0x01068c (0x2170 bytes) 000381 1790896811.601183 punch from 0x031455 to 0x034b34 (0x36e0 bytes) 000382 1790896811.606520 read 0x031d99 thru 0x03aecc (0x9134 bytes) 000384 1790896811.614924 punch from 0x0357c1 to 0x03aecc (0x570c bytes) 000385 1790896811.620627 write 0x026f91 thru 0x036025 (0xf095 bytes) 000386 1790896811.631043 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3aecd): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aecd): Mirror 9: 0xad2baa8d (stale) Mirror 11: 0xad2baa8d (stale) Mirror 12: 0xad2baa8d (stale) Mirror 13: 0xd108afb9 (stale) CRC-32 checksum value for chunk [0, 0x3aecd): Mirror 1: 0x10514ec0 Mirror 14: 0x10514ec0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aecd: skipped 000387 1790896812.913836 mapread 0x0255cb thru 0x02a223 (0x4c59 bytes) 000388 1790896812.939232 mapread 0x01ef63 thru 0x02d1f2 (0xe290 bytes) 000389 1790896812.948995 punch from 0x00c23d to 0x016fb6 (0xad7a bytes) 000390 1790896813.022431 punch from 0x007083 to 0x00c698 (0x5616 bytes) 000391 1790896813.028241 mapwrite 0x0258ae thru 0x034085 (0xe7d8 bytes) 000392 1790896813.096715 read 0x0005a2 thru 0x00d9cd (0xd42c bytes) 000393 1790896813.105038 punch from 0x02fb77 to 0x035d0d (0x6197 bytes) 000394 1790896813.110103 punch from 0x0186a9 to 0x021c1c (0x9574 bytes) 000395 1790896813.117581 read 0x008f0d thru 0x0187b8 (0xf8ac bytes) 000396 1790896813.162699 write 0x03a65a thru 0x03ffff (0x59a6 bytes) 000398 1790896813.167196 read 0x00cd56 thru 0x01288a (0x5b35 bytes) 000399 1790896813.168314 trunc from 0x040000 to 0x015336 (0x2acc9 bytes) 000400 1790896813.212794 mapread 0x014f7a thru 0x015336 (0x3bd bytes) 000401 1790896813.215255 punch from 0x000482 to 0x003616 (0x3195 bytes) 000402 1790896813.220761 mapwrite 0x036e4d thru 0x03ffff (0x91b3 bytes) 000403 1790896813.346892 punch from 0x010109 to 0x01c3b1 (0xc2a9 bytes) 000405 1790896813.361214 mapread 0x00bd89 thru 0x019910 (0xdb88 bytes) 000406 1790896813.454921 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x441ef6c9 (stale) Mirror 11: 0x441ef6c9 (stale) Mirror 12: 0x441ef6c9 (stale) Mirror 13: 0xff432f9d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x441ef6c9 (stale) Mirror 11: 0x441ef6c9 (stale) Mirror 12: 0x441ef6c9 (stale) Mirror 13: 0xff432f9d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000407 1790896813.918435 MIRROR_RESYNC 5 mirrors 000408 1790896815.328834 read 0x033a99 thru 0x039a2e (0x5f96 bytes) 000409 1790896815.337033 mapread 0x003ddd thru 0x00c35f (0x8583 bytes) 000411 1790896815.375817 trunc from 0x040000 to 0x0241b2 (0x1be4d bytes) 000414 1790896815.460554 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x241b3): Mirror 9: 0x161aa3f1 (stale) Mirror 11: 0x161aa3f1 (stale) Mirror 12: 0x161aa3f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x241b3): Mirror 9: 0x161aa3f1 (stale) Mirror 11: 0x161aa3f1 (stale) Mirror 12: 0x161aa3f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x241b3: skipped 000415 1790896815.902698 write 0x0301c6 thru 0x03748d (0x72c8 bytes) 000416 1790896815.911188 write 0x008d8e thru 0x00d5e0 (0x4853 bytes) 000419 1790896815.927293 mapwrite 0x02e83f thru 0x035385 (0x6b47 bytes) 000420 1790896816.074022 mapwrite 0x0138bb thru 0x01ece5 (0xb42b bytes) 000421 1790896816.162363 mapwrite 0x003390 thru 0x00e2ba (0xaf2b bytes) 000422 1790896816.243135 write 0x006d67 thru 0x00d241 (0x64db bytes) 000426 1790896816.252555 write 0x03ca74 thru 0x03ffff (0x358c bytes) 000427 1790896816.257643 punch from 0x0060cc to 0x00d73d (0x7672 bytes) 000430 1790896816.274210 punch from 0x00d30b to 0x00d3f9 (0xef bytes) 000431 1790896816.284250 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x389fb7f4 (stale) Mirror 11: 0x389fb7f4 (stale) Mirror 12: 0x389fb7f4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbe925de1 Mirror 13: 0xbe925de1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000432 1790896817.417175 trunc from 0x040000 to 0x0278ba (0x18745 bytes) 000433 1790896817.521956 write 0x036bb2 thru 0x03a6b6 (0x3b05 bytes) 000434 1790896817.523947 mapwrite 0x00d65d thru 0x011e5c (0x4800 bytes) 000438 1790896817.588201 trunc from 0x03a6b7 to 0x0232ad (0x17409 bytes) 000440 1790896817.606398 write 0x039a3c thru 0x03ce23 (0x33e8 bytes) 000441 1790896817.608434 punch from 0x02cf0b to 0x032363 (0x5459 bytes) 000442 1790896817.619077 mapwrite 0x014f8e thru 0x01aea5 (0x5f18 bytes) 000443 1790896817.679122 mapwrite 0x03dada thru 0x03ffff (0x2526 bytes) 000444 1790896817.761558 trunc from 0x040000 to 0x036b2e (0x94d1 bytes) 000445 1790896817.792117 read 0x012981 thru 0x021ad1 (0xf151 bytes) 000448 1790896817.805088 read 0x0004ac thru 0x007ac7 (0x761c bytes) 000449 1790896817.826190 punch from 0x020493 to 0x026351 (0x5ebf bytes) 000450 1790896817.840758 trunc from 0x036b2f to 0x02bdc5 (0xad69 bytes) 000451 1790896817.861963 read 0x00ce10 thru 0x013aa3 (0x6c94 bytes) 000453 1790896817.863388 trunc from 0x02bdc6 to 0x03bf6b (0x101a6 bytes) 000454 1790896817.887260 mapwrite 0x02359f thru 0x025bc2 (0x2624 bytes) 000455 1790896817.971737 mapwrite 0x007ca7 thru 0x0097b6 (0x1b10 bytes) 000456 1790896818.018341 trunc from 0x03bf6c to 0x01ce45 (0x1f126 bytes) 000457 1790896818.039003 read 0x013d00 thru 0x01ce45 (0x9146 bytes) 000458 1790896818.040748 write 0x03f55f thru 0x03ffff (0xaa1 bytes) 000459 1790896818.050583 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x389fb7f4 (stale) Mirror 11: 0x389fb7f4 (stale) Mirror 12: 0x389fb7f4 (stale) Mirror 13: 0xbe925de1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9a30024c Mirror 14: 0x9a30024c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000460 1790896819.175627 punch from 0x00f672 to 0x01f0b2 (0xfa41 bytes) 000461 1790896819.246736 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x389fb7f4 (stale) Mirror 11: 0x389fb7f4 (stale) Mirror 12: 0x389fb7f4 (stale) Mirror 13: 0xbe925de1 (stale) Mirror 14: 0x9a30024c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd1c92173 Mirror 15: 0xd1c92173 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000463 1790896820.639826 mapwrite 0x00adcc thru 0x0171bc (0xc3f1 bytes) 000465 1790896821.044520 read 0x022634 thru 0x022993 (0x360 bytes) 000466 1790896821.058839 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x389fb7f4 (stale) Mirror 11: 0x389fb7f4 (stale) Mirror 12: 0x389fb7f4 (stale) Mirror 13: 0xbe925de1 (stale) Mirror 14: 0x9a30024c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x389fb7f4 (stale) Mirror 11: 0x389fb7f4 (stale) Mirror 12: 0x389fb7f4 (stale) Mirror 13: 0xbe925de1 (stale) Mirror 14: 0x9a30024c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000471 1790896821.538335 read 0x00fcf7 thru 0x011250 (0x155a bytes) 000472 1790896821.544348 mapread 0x035ea8 thru 0x03f4d9 (0x9632 bytes) 000473 1790896821.560462 write 0x0048a6 thru 0x009d44 (0x549f bytes) 000474 1790896821.571843 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x389fb7f4 (stale) Mirror 11: 0x389fb7f4 (stale) Mirror 12: 0x389fb7f4 (stale) Mirror 13: 0xbe925de1 (stale) Mirror 14: 0x9a30024c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x75e41913 Mirror 15: 0x75e41913 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000476 1790896823.049972 punch from 0x027660 to 0x02b812 (0x41b3 bytes) 000478 1790896823.353210 write 0x00bc73 thru 0x00f878 (0x3c06 bytes) 000479 1790896823.374017 write 0x03d10c thru 0x03ffff (0x2ef4 bytes) 000480 1790896823.391419 punch from 0x02e7f4 to 0x0367f9 (0x8006 bytes) 000481 1790896823.439383 trunc from 0x040000 to 0x033090 (0xcf6f bytes) 000482 1790896823.500503 mapread 0x0068b4 thru 0x014767 (0xdeb4 bytes) 000484 1790896823.577525 mapread 0x0089a3 thru 0x014ba0 (0xc1fe bytes) 000485 1790896823.584437 punch from 0x009379 to 0x00a497 (0x111f bytes) 000486 1790896823.597391 write 0x02e7a1 thru 0x0398e5 (0xb145 bytes) 000487 1790896823.611835 punch from 0x006d63 to 0x00ad15 (0x3fb3 bytes) 000488 1790896823.669015 mapwrite 0x013028 thru 0x01c8d6 (0x98af bytes) 000490 1790896823.841567 mapwrite 0x00fb02 thru 0x01b69e (0xbb9d bytes) 000491 1790896823.987268 write 0x00504e thru 0x007ffd (0x2fb0 bytes) 000493 1790896823.995421 write 0x00e4e4 thru 0x01ba19 (0xd536 bytes) 000494 1790896824.000311 punch from 0x03895f to 0x0398e5 (0xf87 bytes) 000495 1790896824.021000 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x398e6): Mirror 11: 0xc5968371 (stale) Mirror 12: 0xc5968371 (stale) Mirror 13: 0x7c096174 (stale) Mirror 14: 0x73fccaf1 (stale) Mirror 15: 0x7e0e7f40 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398e6): Mirror 11: 0xc5968371 (stale) Mirror 12: 0xc5968371 (stale) Mirror 13: 0x7c096174 (stale) Mirror 14: 0x73fccaf1 (stale) Mirror 15: 0x7e0e7f40 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398e6: skipped 000496 1790896824.683863 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x398e6): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x398e6): Mirror 11: 0xc5968371 (stale) Mirror 12: 0xc5968371 (stale) Mirror 13: 0x7c096174 (stale) Mirror 14: 0x73fccaf1 (stale) Mirror 15: 0x7e0e7f40 (stale) CRC-32 checksum value for chunk [0, 0x398e6): Mirror 1: 0xe732cdf0 Mirror 16: 0xe732cdf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x398e6: skipped 000497 1790896826.176367 write 0x035b1a thru 0x038b59 (0x3040 bytes) 000499 1790896826.349994 punch from 0x009f06 to 0x00f757 (0x5852 bytes) 000502 1790896826.386593 read 0x014bd8 thru 0x0186b2 (0x3adb bytes) 000503 1790896826.403090 punch from 0x02e625 to 0x02eb02 (0x4de bytes) 000505 1790896826.409123 mapwrite 0x0193b4 thru 0x024b59 (0xb7a6 bytes) 000506 1790896826.488411 mapwrite 0x03aaf8 thru 0x03ffff (0x5508 bytes) 000507 1790896826.600354 read 0x03edc8 thru 0x03ffff (0x1238 bytes) 000508 1790896826.601221 write 0x03c09f thru 0x03ffff (0x3f61 bytes) 000510 1790896826.604239 mapread 0x025e14 thru 0x02af6d (0x515a bytes) 000511 1790896826.606370 read 0x032bac thru 0x0348ec (0x1d41 bytes) 000512 1790896826.615063 mapwrite 0x000c42 thru 0x00ed45 (0xe104 bytes) 000513 1790896826.665549 write 0x0361c3 thru 0x03ef0c (0x8d4a bytes) 000515 1790896826.668018 trunc from 0x040000 to 0x013545 (0x2caba bytes) 000516 1790896826.683831 trunc from 0x013546 to 0x0196c0 (0x617b bytes) 000518 1790896826.704606 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x196c1): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x196c1): Mirror 11: 0x4dcf70a9 (stale) Mirror 12: 0x4dcf70a9 (stale) Mirror 13: 0x304ca81a (stale) Mirror 14: 0x28de8a09 (stale) Mirror 15: 0x7bac6507 (stale) Mirror 16: 0x7da3a216 (stale) CRC-32 checksum value for chunk [0, 0x196c1): Mirror 1: 0xb9aec771 Mirror 17: 0xb9aec771 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x196c1: skipped 000519 1790896828.067729 punch from 0x0156a7 to 0x0196c0 (0x401a bytes) 000520 1790896828.283120 mapwrite 0x02c04f thru 0x03a099 (0xe04b bytes) 000521 1790896828.520428 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3a09a): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a09a): Mirror 11: 0xae374220 (stale) Mirror 12: 0xae374220 (stale) Mirror 13: 0xa76d5072 (stale) Mirror 14: 0x4ed266c9 (stale) Mirror 15: 0x4da6f930 (stale) Mirror 16: 0xb5a07d9b (stale) Mirror 17: 0xd0ca7e10 (stale) CRC-32 checksum value for chunk [0, 0x3a09a): Mirror 1: 0x59d1a956 Mirror 18: 0x59d1a956 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a09a: skipped 000522 1790896829.921565 write 0x0093b0 thru 0x013fcc (0xac1d bytes) 000524 1790896830.023248 write 0x03d887 thru 0x03ffff (0x2779 bytes) 000525 1790896830.024416 mapread 0x00c0fa thru 0x00f6bd (0x35c4 bytes) 000527 1790896830.027023 mapread 0x0288ad thru 0x036148 (0xd89c bytes) 000529 1790896830.052382 read 0x033a5f thru 0x03c06b (0x860d bytes) 000530 1790896830.058192 punch from 0x019322 to 0x01f111 (0x5df0 bytes) 000531 1790896830.075340 mapread 0x00131e thru 0x00de2b (0xcb0e bytes) 000532 1790896830.089637 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x389fb7f4 (stale) Mirror 12: 0x389fb7f4 (stale) Mirror 13: 0xbe925de1 (stale) Mirror 14: 0x9a30024c (stale) Mirror 15: 0x75e41913 (stale) Mirror 16: 0xe869c600 (stale) Mirror 17: 0xf8cf8ee5 (stale) Mirror 18: 0x7a91daa1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2125e3a4 Mirror 19: 0x2125e3a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000534 1790896831.774963 trunc from 0x040000 to 0x034936 (0xb6c9 bytes) 000535 1790896831.973910 mapwrite 0x01cdd9 thru 0x02bf59 (0xf181 bytes) 000536 1790896832.086649 mapread 0x02935e thru 0x031929 (0x85cc bytes) 000538 1790896832.102949 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x34937): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34937): Mirror 11: 0xa78b8ab4 (stale) Mirror 12: 0xa78b8ab4 (stale) Mirror 13: 0xca03e2d1 (stale) Mirror 14: 0x32078e55 (stale) Mirror 15: 0x4dfb678d (stale) Mirror 16: 0x2126aaca (stale) Mirror 17: 0xb967e4e1 (stale) Mirror 18: 0xab94ff94 (stale) Mirror 19: 0xb4035c92 (stale) CRC-32 checksum value for chunk [0, 0x34937): Mirror 1: 0x6c54033a Mirror 20: 0x6c54033a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34937: skipped 000539 1790896833.455700 mapread 0x011d72 thru 0x017783 (0x5a12 bytes) 000540 1790896833.488938 mapread 0x00ff62 thru 0x01c1db (0xc27a bytes) 000541 1790896833.516578 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x34937): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34937): Mirror 11: 0xa78b8ab4 (stale) Mirror 12: 0xa78b8ab4 (stale) Mirror 13: 0xca03e2d1 (stale) Mirror 14: 0x32078e55 (stale) Mirror 15: 0x4dfb678d (stale) Mirror 16: 0x2126aaca (stale) Mirror 17: 0xb967e4e1 (stale) Mirror 18: 0xab94ff94 (stale) Mirror 19: 0xb4035c92 (stale) CRC-32 checksum value for chunk [0, 0x34937): Mirror 1: 0x6c54033a Mirror 20: 0x6c54033a Mirror 21: 0x6c54033a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34937: skipped 000542 1790896834.948596 MIRROR_RESYNC 12 mirrors 000543 1790896836.312534 write 0x0254e8 thru 0x0275f0 (0x2109 bytes) 000544 1790896836.385393 read 0x019cfc thru 0x028a2f (0xed34 bytes) 000546 1790896836.394436 read 0x00f4c0 thru 0x0149c1 (0x5502 bytes) 000548 1790896836.405725 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x34937): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34937): Mirror 11: 0x6c54033a (stale) Mirror 12: 0x6c54033a (stale) Mirror 13: 0x6c54033a (stale) Mirror 14: 0x6c54033a (stale) Mirror 15: 0x6c54033a (stale) Mirror 16: 0x6c54033a (stale) Mirror 17: 0x6c54033a (stale) Mirror 18: 0x6c54033a (stale) Mirror 19: 0x6c54033a (stale) Mirror 20: 0x6c54033a (stale) Mirror 21: 0x6c54033a (stale) CRC-32 checksum value for chunk [0, 0x34937): Mirror 1: 0xc78eebf Mirror 22: 0xc78eebf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34937: skipped 000549 1790896837.468121 trunc from 0x034937 to 0x0185b7 (0x1c37f bytes) 000550 1790896837.582306 mapread 0x012cd9 thru 0x0185b7 (0x58df bytes) 000551 1790896837.599638 trunc from 0x0185b8 to 0x0347a9 (0x1c1f2 bytes) 000553 1790896837.623209 mapread 0x002e69 thru 0x011e86 (0xf01e bytes) 000557 1790896837.635544 read 0x00226f thru 0x006aae (0x4840 bytes) 000558 1790896837.636512 read 0x00e08a thru 0x01c7f6 (0xe76d bytes) 000559 1790896837.653976 mapwrite 0x009964 thru 0x00cf07 (0x35a4 bytes) 000561 1790896837.716259 trunc from 0x0347aa to 0x0114ba (0x232ef bytes) 000562 1790896837.748646 trunc from 0x0114bb to 0x03b228 (0x29d6e bytes) 000564 1790896837.783250 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3b229): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b229): Mirror 11: 0x2ebd291b (stale) Mirror 12: 0x2ebd291b (stale) Mirror 13: 0x2ebd291b (stale) Mirror 14: 0x2ebd291b (stale) Mirror 15: 0x2ebd291b (stale) Mirror 16: 0x2ebd291b (stale) Mirror 17: 0x2ebd291b (stale) Mirror 18: 0x2ebd291b (stale) Mirror 19: 0x2ebd291b (stale) Mirror 20: 0x2ebd291b (stale) Mirror 21: 0x2ebd291b (stale) Mirror 22: 0xb8a84c85 (stale) CRC-32 checksum value for chunk [0, 0x3b229): Mirror 1: 0x11a6b3e9 Mirror 23: 0x11a6b3e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b229: skipped 000568 1790896839.866735 read 0x027102 thru 0x036165 (0xf064 bytes) 000570 1790896839.924791 mapread 0x035970 thru 0x03b228 (0x58b9 bytes) 000571 1790896839.980924 write 0x023f4a thru 0x02aff4 (0x70ab bytes) 000575 1790896840.157071 write 0x01faec thru 0x0205ce (0xae3 bytes) 000576 1790896840.174061 mapread 0x01e8b6 thru 0x02b3f1 (0xcb3c bytes) 000577 1790896840.217538 punch from 0x01e019 to 0x0255dd (0x75c5 bytes) 000578 1790896840.238913 mapwrite 0x02e558 thru 0x0311d0 (0x2c79 bytes) 000579 1790896840.300896 trunc from 0x03b229 to 0x0379dd (0x384b bytes) 000581 1790896840.334337 write 0x02cad3 thru 0x03702d (0xa55b bytes) 000585 1790896840.344791 mapwrite 0x009cb0 thru 0x015386 (0xb6d7 bytes) 000586 1790896840.452269 write 0x003dca thru 0x00d46c (0x96a3 bytes) 000587 1790896840.453514 punch from 0x036a88 to 0x0379dd (0xf56 bytes) 000588 1790896840.504645 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x379de): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x379de): Mirror 11: 0xf02bbf97 (stale) Mirror 12: 0xf02bbf97 (stale) Mirror 13: 0xf02bbf97 (stale) Mirror 14: 0xf02bbf97 (stale) Mirror 15: 0xf02bbf97 (stale) Mirror 16: 0xf02bbf97 (stale) Mirror 17: 0xf02bbf97 (stale) Mirror 18: 0xf02bbf97 (stale) Mirror 19: 0xf02bbf97 (stale) Mirror 20: 0xf02bbf97 (stale) Mirror 21: 0xf02bbf97 (stale) Mirror 22: 0xf4dc4eef (stale) Mirror 23: 0x9c9a0042 (stale) CRC-32 checksum value for chunk [0, 0x379de): Mirror 1: 0x61941267 Mirror 24: 0x61941267 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x379de: skipped 000590 1790896842.222805 trunc from 0x0379de to 0x0225b8 (0x15425 bytes) 000591 1790896842.333079 read 0x003a4e thru 0x00fc19 (0xc1cc bytes) 000592 1790896842.343516 write 0x007181 thru 0x00917b (0x1ffb bytes) 000593 1790896842.344607 mapread 0x01dde1 thru 0x01f589 (0x17a9 bytes) 000594 1790896842.361455 mapwrite 0x020da7 thru 0x02c57d (0xb7d7 bytes) 000597 1790896842.456451 mapread 0x01c801 thru 0x022e81 (0x6681 bytes) 000598 1790896842.457836 punch from 0x00d29a to 0x013c3a (0x69a1 bytes) 000599 1790896842.463041 read 0x00967c thru 0x01967b (0x10000 bytes) 000600 1790896842.472400 punch from 0x01a539 to 0x01e102 (0x3bca bytes) 000601 1790896842.477381 write 0x005e05 thru 0x012372 (0xc56e bytes) 000604 1790896842.480983 mapread 0x01b1d5 thru 0x01cf29 (0x1d55 bytes) 000606 1790896842.491490 read 0x02230a thru 0x027513 (0x520a bytes) 000607 1790896842.492362 trunc from 0x02c57e to 0x03891e (0xc3a1 bytes) 000608 1790896842.511333 read 0x02296f thru 0x02b6fe (0x8d90 bytes) 000610 1790896842.512547 mapread 0x00dd11 thru 0x013f80 (0x6270 bytes) 000612 1790896842.518186 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3891f): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3891f): Mirror 11: 0x2c977a80 (stale) Mirror 12: 0x2c977a80 (stale) Mirror 13: 0x2c977a80 (stale) Mirror 14: 0x2c977a80 (stale) Mirror 15: 0x2c977a80 (stale) Mirror 16: 0x2c977a80 (stale) Mirror 17: 0x2c977a80 (stale) Mirror 18: 0x2c977a80 (stale) Mirror 19: 0x2c977a80 (stale) Mirror 20: 0x2c977a80 (stale) Mirror 21: 0x2c977a80 (stale) Mirror 22: 0xc90d300c (stale) Mirror 23: 0xd1cac2c (stale) Mirror 24: 0xc35c29c1 (stale) CRC-32 checksum value for chunk [0, 0x3891f): Mirror 1: 0x45fc7479 Mirror 25: 0x45fc7479 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3891f: skipped 000613 1790896844.181695 write 0x027b3c thru 0x02b238 (0x36fd bytes) 000614 1790896844.324312 mapread 0x02a2d8 thru 0x02e5ca (0x42f3 bytes) 000620 1790896844.350946 trunc from 0x03891f to 0x02e6b4 (0xa26a bytes) 000621 1790896844.377995 mapread 0x005736 thru 0x00ca3a (0x7305 bytes) 000623 1790896844.397149 MIRROR_RESYNC 16 mirrors 000624 1790896846.643008 mapwrite 0x00c238 thru 0x012896 (0x665f bytes) 000625 1790896846.825014 mapwrite 0x03692d thru 0x03e7d6 (0x7eaa bytes) 000626 1790896846.958712 trunc from 0x03e7d7 to 0x03fc71 (0x149b bytes) 000628 1790896846.974424 write 0x02a32d thru 0x035751 (0xb425 bytes) 000629 1790896846.983825 write 0x0192bd thru 0x023fb9 (0xacfd bytes) 000631 1790896846.995392 mapread 0x016f99 thru 0x0254ce (0xe536 bytes) 000632 1790896847.010442 read 0x01e6c4 thru 0x0214dc (0x2e19 bytes) 000633 1790896847.011068 mapread 0x02c5e3 thru 0x02c9da (0x3f8 bytes) 000634 1790896847.019874 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc72): Mirror 12: 0x19a1b7fe (stale) Mirror 13: 0x19a1b7fe (stale) Mirror 14: 0x19a1b7fe (stale) Mirror 15: 0x19a1b7fe (stale) Mirror 16: 0x19a1b7fe (stale) Mirror 17: 0x19a1b7fe (stale) Mirror 18: 0x19a1b7fe (stale) Mirror 19: 0x19a1b7fe (stale) Mirror 20: 0x19a1b7fe (stale) Mirror 21: 0x19a1b7fe (stale) Mirror 22: 0x19a1b7fe (stale) Mirror 23: 0x19a1b7fe (stale) Mirror 24: 0x19a1b7fe (stale) Mirror 25: 0x19a1b7fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc72): Mirror 12: 0x19a1b7fe (stale) Mirror 13: 0x19a1b7fe (stale) Mirror 14: 0x19a1b7fe (stale) Mirror 15: 0x19a1b7fe (stale) Mirror 16: 0x19a1b7fe (stale) Mirror 17: 0x19a1b7fe (stale) Mirror 18: 0x19a1b7fe (stale) Mirror 19: 0x19a1b7fe (stale) Mirror 20: 0x19a1b7fe (stale) Mirror 21: 0x19a1b7fe (stale) Mirror 22: 0x19a1b7fe (stale) Mirror 23: 0x19a1b7fe (stale) Mirror 24: 0x19a1b7fe (stale) Mirror 25: 0x19a1b7fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc72: skipped 000635 1790896847.795884 punch from 0x01ab3a to 0x02818b (0xd652 bytes) 000636 1790896847.809062 mapread 0x031529 thru 0x0340ef (0x2bc7 bytes) 000638 1790896847.837880 read 0x005596 thru 0x00a6d6 (0x5141 bytes) 000639 1790896847.846787 mapwrite 0x03f30d thru 0x03f726 (0x41a bytes) 000640 1790896848.006980 mapwrite 0x004d1b thru 0x00e6be (0x99a4 bytes) 000642 1790896848.109895 read 0x027c3f thru 0x035f31 (0xe2f3 bytes) 000644 1790896848.134903 mapwrite 0x008c54 thru 0x00d041 (0x43ee bytes) 000645 1790896848.200053 mapwrite 0x01813b thru 0x023876 (0xb73c bytes) 000646 1790896848.258573 mapread 0x021cbc thru 0x02c5c9 (0xa90e bytes) 000647 1790896848.259900 mapread 0x039cd3 thru 0x03fc71 (0x5f9f bytes) 000648 1790896848.261224 read 0x029b5d thru 0x02fef3 (0x6397 bytes) 000651 1790896848.263386 write 0x007e64 thru 0x008a5b (0xbf8 bytes) 000652 1790896848.264443 read 0x01acd4 thru 0x021b4a (0x6e77 bytes) 000655 1790896848.266765 trunc from 0x03fc72 to 0x019405 (0x2686c bytes) 000657 1790896848.282878 trunc from 0x019406 to 0x034776 (0x1b371 bytes) 000658 1790896848.298917 trunc from 0x034777 to 0x00163f (0x33137 bytes) 000659 1790896848.316607 write 0x017472 thru 0x01b938 (0x44c7 bytes) 000661 1790896848.319745 mapread 0x0058cd thru 0x00ca4f (0x7183 bytes) 000662 1790896848.327779 punch from 0x015a93 to 0x01b938 (0x5ea6 bytes) 000663 1790896848.340893 punch from 0x0138f4 to 0x01a2de (0x69eb bytes) 000666 1790896848.348444 mapwrite 0x03d6dc thru 0x03ffff (0x2924 bytes) 000667 1790896848.412616 mapwrite 0x017a10 thru 0x023a82 (0xc073 bytes) 000668 1790896848.526774 mapwrite 0x00b96f thru 0x01a728 (0xedba bytes) 000669 1790896848.602636 mapwrite 0x00c3e4 thru 0x010497 (0x40b4 bytes) 000670 1790896848.659967 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x26876982 (stale) Mirror 13: 0x26876982 (stale) Mirror 14: 0x26876982 (stale) Mirror 15: 0x26876982 (stale) Mirror 16: 0x26876982 (stale) Mirror 17: 0x26876982 (stale) Mirror 18: 0x26876982 (stale) Mirror 19: 0x26876982 (stale) Mirror 20: 0x26876982 (stale) Mirror 21: 0x26876982 (stale) Mirror 22: 0x26876982 (stale) Mirror 23: 0x26876982 (stale) Mirror 24: 0x26876982 (stale) Mirror 25: 0x26876982 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x88ada263 Mirror 26: 0x88ada263 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000671 1790896850.256577 mapwrite 0x009953 thru 0x01988e (0xff3c bytes) 000675 1790896850.465699 mapwrite 0x0279b2 thru 0x02ff39 (0x8588 bytes) 000676 1790896850.531764 read 0x00d901 thru 0x00e9fe (0x10fe bytes) 000677 1790896850.535619 MIRROR_RESYNC 16 mirrors 000678 1790896854.228316 mapwrite 0x03c0c2 thru 0x03ffff (0x3f3e bytes) 000679 1790896854.419949 mapwrite 0x02ccd9 thru 0x032f95 (0x62bd bytes) 000680 1790896854.537634 read 0x0062f5 thru 0x00a091 (0x3d9d bytes) 000681 1790896854.547021 trunc from 0x040000 to 0x02fe9f (0x10160 bytes) 000685 1790896854.588720 mapread 0x01fd09 thru 0x02b506 (0xb7fe bytes) 000686 1790896854.602706 trunc from 0x02fea0 to 0x003a56 (0x2c449 bytes) 000687 1790896854.652473 mapread 0x000c83 thru 0x003a56 (0x2dd4 bytes) 000688 1790896854.667032 mapwrite 0x018725 thru 0x022778 (0xa054 bytes) 000689 1790896854.816059 write 0x01cf56 thru 0x021aad (0x4b58 bytes) 000691 1790896854.817579 mapwrite 0x00ab0f thru 0x0113ac (0x689e bytes) 000692 1790896854.877286 trunc from 0x022779 to 0x023772 (0xffa bytes) 000693 1790896854.891994 trunc from 0x023773 to 0x02d8d4 (0xa162 bytes) 000694 1790896854.911579 mapwrite 0x023833 thru 0x02eaab (0xb279 bytes) 000695 1790896855.002915 trunc from 0x02eaac to 0x0220b2 (0xc9f9 bytes) 000697 1790896855.023457 read 0x0182f1 thru 0x01cfbd (0x4ccd bytes) 000698 1790896855.024211 mapwrite 0x00a66e thru 0x015139 (0xaacc bytes) 000699 1790896855.098001 trunc from 0x0220b3 to 0x00f771 (0x12941 bytes) 000700 1790896855.122849 MIRROR_SPLIT mirror 10 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf772): Mirror 12: 0xefe08ed8 (stale) Mirror 13: 0xefe08ed8 (stale) Mirror 14: 0xefe08ed8 (stale) Mirror 15: 0xefe08ed8 (stale) Mirror 16: 0xefe08ed8 (stale) Mirror 17: 0xefe08ed8 (stale) Mirror 18: 0xefe08ed8 (stale) Mirror 19: 0xefe08ed8 (stale) Mirror 20: 0xefe08ed8 (stale) Mirror 22: 0xefe08ed8 (stale) Mirror 23: 0xefe08ed8 (stale) Mirror 24: 0xefe08ed8 (stale) Mirror 25: 0xefe08ed8 (stale) Mirror 26: 0xefe08ed8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf772): Mirror 12: 0xefe08ed8 (stale) Mirror 13: 0xefe08ed8 (stale) Mirror 14: 0xefe08ed8 (stale) Mirror 15: 0xefe08ed8 (stale) Mirror 16: 0xefe08ed8 (stale) Mirror 17: 0xefe08ed8 (stale) Mirror 18: 0xefe08ed8 (stale) Mirror 19: 0xefe08ed8 (stale) Mirror 20: 0xefe08ed8 (stale) Mirror 22: 0xefe08ed8 (stale) Mirror 23: 0xefe08ed8 (stale) Mirror 24: 0xefe08ed8 (stale) Mirror 25: 0xefe08ed8 (stale) Mirror 26: 0xefe08ed8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf772: skipped 000701 1790896855.884510 punch from 0x00229e to 0x00d15e (0xaec1 bytes) 000702 1790896855.889059 mapwrite 0x02c91c thru 0x03abb7 (0xe29c bytes) 000703 1790896856.020772 read 0x00e459 thru 0x01b620 (0xd1c8 bytes) 000704 1790896856.030061 mapread 0x02ae34 thru 0x02d100 (0x22cd bytes) 000705 1790896856.036229 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3abb8): Mirror 12: 0x173f8d55 (stale) Mirror 13: 0x173f8d55 (stale) Mirror 14: 0x173f8d55 (stale) Mirror 15: 0x173f8d55 (stale) Mirror 16: 0x173f8d55 (stale) Mirror 17: 0x173f8d55 (stale) Mirror 18: 0x173f8d55 (stale) Mirror 19: 0x173f8d55 (stale) Mirror 20: 0x173f8d55 (stale) Mirror 22: 0x173f8d55 (stale) Mirror 24: 0x173f8d55 (stale) Mirror 25: 0x173f8d55 (stale) Mirror 26: 0x173f8d55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3abb8): Mirror 12: 0x173f8d55 (stale) Mirror 13: 0x173f8d55 (stale) Mirror 14: 0x173f8d55 (stale) Mirror 15: 0x173f8d55 (stale) Mirror 16: 0x173f8d55 (stale) Mirror 17: 0x173f8d55 (stale) Mirror 18: 0x173f8d55 (stale) Mirror 19: 0x173f8d55 (stale) Mirror 20: 0x173f8d55 (stale) Mirror 22: 0x173f8d55 (stale) Mirror 24: 0x173f8d55 (stale) Mirror 25: 0x173f8d55 (stale) Mirror 26: 0x173f8d55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3abb8: skipped 000706 1790896856.807540 write 0x01ceec thru 0x02114d (0x4262 bytes) 000709 1790896856.817608 punch from 0x02029d to 0x02a271 (0x9fd5 bytes) 000710 1790896856.832405 MIRROR_RESYNC 14 mirrors 000711 1790896860.227398 MIRROR_RESYNC 14 mirrors 000712 1790896861.338919 read 0x00c911 thru 0x01bc46 (0xf336 bytes) 000714 1790896861.346908 punch from 0x009163 to 0x013de9 (0xac87 bytes) 000715 1790896861.572023 mapread 0x0151f7 thru 0x01a426 (0x5230 bytes) 000717 1790896861.605604 trunc from 0x03abb8 to 0x006f53 (0x33c64 bytes) 000718 1790896861.652771 mapread 0x000b6a thru 0x006f53 (0x63ea bytes) 000719 1790896861.672001 punch from 0x002649 to 0x00486b (0x2223 bytes) 000721 1790896861.677385 mapwrite 0x00b85f thru 0x00fb16 (0x42b8 bytes) 000723 1790896861.796479 MIRROR_RESYNC 14 mirrors 000725 1790896864.084071 punch from 0x00c9dc to 0x00fb16 (0x313b bytes) 000726 1790896864.173011 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0xfb17): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfb17): Mirror 12: 0xfb6c5c0 (stale) Mirror 13: 0xfb6c5c0 (stale) Mirror 14: 0xfb6c5c0 (stale) Mirror 15: 0xfb6c5c0 (stale) Mirror 16: 0xfb6c5c0 (stale) Mirror 17: 0xfb6c5c0 (stale) Mirror 18: 0xfb6c5c0 (stale) Mirror 19: 0xfb6c5c0 (stale) Mirror 20: 0xfb6c5c0 (stale) Mirror 22: 0xfb6c5c0 (stale) Mirror 24: 0xfb6c5c0 (stale) Mirror 25: 0xfb6c5c0 (stale) Mirror 26: 0xfb6c5c0 (stale) CRC-32 checksum value for chunk [0, 0xfb17): Mirror 1: 0x8a173454 Mirror 27: 0x8a173454 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfb17: skipped 000728 1790896865.555929 read 0x000f90 thru 0x00a260 (0x92d1 bytes) 000729 1790896865.584742 punch from 0x004d51 to 0x00ee62 (0xa112 bytes) 000730 1790896865.744756 mapwrite 0x000d5d thru 0x00c4b2 (0xb756 bytes) 000732 1790896865.837912 read 0x0028b3 thru 0x00b8d5 (0x9023 bytes) 000733 1790896865.839581 trunc from 0x00fb17 to 0x023f12 (0x143fc bytes) 000734 1790896865.885889 mapread 0x0227af thru 0x023f12 (0x1764 bytes) 000735 1790896865.897305 write 0x028c65 thru 0x02d90d (0x4ca9 bytes) 000736 1790896865.899729 read 0x01311a thru 0x01d65f (0xa546 bytes) 000737 1790896865.900429 trunc from 0x02d90e to 0x02a6f1 (0x321c bytes) 000739 1790896865.943728 punch from 0x020942 to 0x02205b (0x171a bytes) 000741 1790896865.967135 write 0x022c72 thru 0x031a71 (0xee00 bytes) 000742 1790896865.980058 punch from 0x019642 to 0x01e81f (0x51de bytes) 000743 1790896866.022518 write 0x030e3c thru 0x0353b2 (0x4577 bytes) 000745 1790896866.030288 read 0x0290db thru 0x0298ba (0x7e0 bytes) 000746 1790896866.031076 trunc from 0x0353b3 to 0x01dfea (0x173c8 bytes) 000748 1790896866.059572 trunc from 0x01dfeb to 0x00797f (0x1666b bytes) 000749 1790896866.093960 trunc from 0x007980 to 0x03cfea (0x3566b bytes) 000750 1790896866.120578 mapwrite 0x012e9d thru 0x01787f (0x49e3 bytes) 000751 1790896866.242543 read 0x0317fb thru 0x03cfea (0xb7f0 bytes) 000752 1790896866.243395 punch from 0x003a5f to 0x011b60 (0xe102 bytes) 000754 1790896866.252007 punch from 0x0116e5 to 0x012cf5 (0x1611 bytes) 000755 1790896866.267282 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cfeb): Mirror 12: 0x3fe74fe3 (stale) Mirror 14: 0x3fe74fe3 (stale) Mirror 15: 0x3fe74fe3 (stale) Mirror 16: 0x3fe74fe3 (stale) Mirror 17: 0x3fe74fe3 (stale) Mirror 18: 0x3fe74fe3 (stale) Mirror 19: 0x3fe74fe3 (stale) Mirror 20: 0x3fe74fe3 (stale) Mirror 22: 0x3fe74fe3 (stale) Mirror 24: 0x3fe74fe3 (stale) Mirror 25: 0x3fe74fe3 (stale) Mirror 26: 0x3fe74fe3 (stale) Mirror 27: 0x102f5012 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cfeb): Mirror 12: 0x3fe74fe3 (stale) Mirror 14: 0x3fe74fe3 (stale) Mirror 15: 0x3fe74fe3 (stale) Mirror 16: 0x3fe74fe3 (stale) Mirror 17: 0x3fe74fe3 (stale) Mirror 18: 0x3fe74fe3 (stale) Mirror 19: 0x3fe74fe3 (stale) Mirror 20: 0x3fe74fe3 (stale) Mirror 22: 0x3fe74fe3 (stale) Mirror 24: 0x3fe74fe3 (stale) Mirror 25: 0x3fe74fe3 (stale) Mirror 26: 0x3fe74fe3 (stale) Mirror 27: 0x102f5012 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cfeb: skipped 000756 1790896867.046438 punch from 0x036019 to 0x03cfea (0x6fd2 bytes) 000758 1790896867.052972 mapread 0x016894 thru 0x01dbba (0x7327 bytes) 000759 1790896867.082015 mapread 0x0060f8 thru 0x011b4a (0xba53 bytes) 000760 1790896867.102381 mapwrite 0x038dfc thru 0x03ffff (0x7204 bytes) 000761 1790896867.218370 punch from 0x01a7f5 to 0x028e2f (0xe63b bytes) 000764 1790896867.227319 trunc from 0x040000 to 0x02cbb8 (0x13447 bytes) 000766 1790896867.250328 mapwrite 0x01c742 thru 0x029468 (0xcd27 bytes) 000768 1790896867.350542 read 0x00c40f thru 0x01b751 (0xf343 bytes) 000769 1790896867.352319 mapread 0x01577a thru 0x022454 (0xccdb bytes) 000770 1790896867.354548 trunc from 0x02cbb9 to 0x026381 (0x6837 bytes) 000772 1790896867.396949 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x26382): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26382): Mirror 12: 0xd4760eca (stale) Mirror 14: 0xd4760eca (stale) Mirror 15: 0xd4760eca (stale) Mirror 16: 0xd4760eca (stale) Mirror 17: 0xd4760eca (stale) Mirror 18: 0xd4760eca (stale) Mirror 19: 0xd4760eca (stale) Mirror 20: 0xd4760eca (stale) Mirror 22: 0xd4760eca (stale) Mirror 24: 0xd4760eca (stale) Mirror 25: 0xd4760eca (stale) Mirror 26: 0xd4760eca (stale) Mirror 27: 0xdf3f2be0 (stale) CRC-32 checksum value for chunk [0, 0x26382): Mirror 1: 0x3442c45f Mirror 28: 0x3442c45f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26382: skipped 000775 1790896869.060953 trunc from 0x026382 to 0x025213 (0x116e bytes) 000776 1790896869.215245 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25214): Mirror 12: 0x9b371fee (stale) Mirror 14: 0x9b371fee (stale) Mirror 15: 0x9b371fee (stale) Mirror 16: 0x9b371fee (stale) Mirror 18: 0x9b371fee (stale) Mirror 19: 0x9b371fee (stale) Mirror 20: 0x9b371fee (stale) Mirror 22: 0x9b371fee (stale) Mirror 24: 0x9b371fee (stale) Mirror 25: 0x9b371fee (stale) Mirror 26: 0x9b371fee (stale) Mirror 27: 0xe6fc4659 (stale) Mirror 28: 0x703fbfa7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25214): Mirror 12: 0x9b371fee (stale) Mirror 14: 0x9b371fee (stale) Mirror 15: 0x9b371fee (stale) Mirror 16: 0x9b371fee (stale) Mirror 18: 0x9b371fee (stale) Mirror 19: 0x9b371fee (stale) Mirror 20: 0x9b371fee (stale) Mirror 22: 0x9b371fee (stale) Mirror 24: 0x9b371fee (stale) Mirror 25: 0x9b371fee (stale) Mirror 26: 0x9b371fee (stale) Mirror 27: 0xe6fc4659 (stale) Mirror 28: 0x703fbfa7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25214: skipped 000778 1790896869.781400 trunc from 0x025214 to 0x005711 (0x1fb02 bytes) 000779 1790896869.798444 trunc from 0x005712 to 0x02ff7a (0x2a869 bytes) 000780 1790896869.830410 mapread 0x01c2ef thru 0x025676 (0x9388 bytes) 000781 1790896869.857474 read 0x012460 thru 0x017f25 (0x5ac6 bytes) 000784 1790896869.877934 read 0x00a561 thru 0x01a4a8 (0xff48 bytes) 000787 1790896869.894714 read 0x01131b thru 0x014a0a (0x36f0 bytes) 000788 1790896869.895268 write 0x03129c thru 0x03e770 (0xd4d5 bytes) 000790 1790896869.913126 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3e771): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e771): Mirror 12: 0xd3a27004 (stale) Mirror 14: 0xd3a27004 (stale) Mirror 15: 0xd3a27004 (stale) Mirror 16: 0xd3a27004 (stale) Mirror 18: 0xd3a27004 (stale) Mirror 19: 0xd3a27004 (stale) Mirror 20: 0xd3a27004 (stale) Mirror 22: 0xd3a27004 (stale) Mirror 24: 0xd3a27004 (stale) Mirror 25: 0xd3a27004 (stale) Mirror 26: 0xd3a27004 (stale) Mirror 27: 0xf0a1b98c (stale) Mirror 28: 0xbc35b70a (stale) CRC-32 checksum value for chunk [0, 0x3e771): Mirror 1: 0xd50ce3ef Mirror 29: 0xd50ce3ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e771: skipped 000791 1790896871.089689 MIRROR_RESYNC 15 mirrors 000793 1790896875.135355 trunc from 0x03e771 to 0x027abd (0x16cb3 bytes) 000794 1790896875.274442 read 0x00cf15 thru 0x016e60 (0x9f4c bytes) 000795 1790896875.327553 read 0x023ddf thru 0x027563 (0x3785 bytes) 000796 1790896875.344267 trunc from 0x027abe to 0x02fce0 (0x8223 bytes) 000798 1790896875.371548 trunc from 0x02fce1 to 0x026a8b (0x9255 bytes) 000799 1790896875.403883 MIRROR_SPLIT mirror 5 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26a8c): Mirror 12: 0x848e665c (stale) Mirror 14: 0x848e665c (stale) Mirror 15: 0x848e665c (stale) Mirror 16: 0x848e665c (stale) Mirror 19: 0x848e665c (stale) Mirror 20: 0x848e665c (stale) Mirror 22: 0x848e665c (stale) Mirror 24: 0x848e665c (stale) Mirror 25: 0x848e665c (stale) Mirror 26: 0x848e665c (stale) Mirror 27: 0x848e665c (stale) Mirror 28: 0x848e665c (stale) Mirror 29: 0x848e665c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a8c): Mirror 12: 0x848e665c (stale) Mirror 14: 0x848e665c (stale) Mirror 15: 0x848e665c (stale) Mirror 16: 0x848e665c (stale) Mirror 19: 0x848e665c (stale) Mirror 20: 0x848e665c (stale) Mirror 22: 0x848e665c (stale) Mirror 24: 0x848e665c (stale) Mirror 25: 0x848e665c (stale) Mirror 26: 0x848e665c (stale) Mirror 27: 0x848e665c (stale) Mirror 28: 0x848e665c (stale) Mirror 29: 0x848e665c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a8c: skipped 000801 1790896876.352768 mapwrite 0x025407 thru 0x02d9ba (0x85b4 bytes) 000802 1790896876.505330 read 0x002f34 thru 0x00722d (0x42fa bytes) 000803 1790896876.515268 write 0x01e68f thru 0x025564 (0x6ed6 bytes) 000804 1790896876.520061 mapwrite 0x00f4b0 thru 0x01eebb (0xfa0c bytes) 000805 1790896876.581552 punch from 0x013e00 to 0x020417 (0xc618 bytes) 000806 1790896876.586745 trunc from 0x02d9bb to 0x01c034 (0x11986 bytes) 000807 1790896876.610236 MIRROR_RESYNC 14 mirrors 000809 1790896878.646149 read 0x01b59e thru 0x01c034 (0xa97 bytes) 000810 1790896878.664165 mapwrite 0x03bde0 thru 0x03ffff (0x4220 bytes) 000811 1790896878.915259 mapwrite 0x008855 thru 0x012d7a (0xa526 bytes) 000812 1790896878.975222 punch from 0x039236 to 0x03ffff (0x6dca bytes) 000813 1790896878.980842 read 0x01c9bd thru 0x025d83 (0x93c7 bytes) 000814 1790896879.000196 mapwrite 0x0331cb thru 0x03a2cf (0x7105 bytes) 000815 1790896879.044036 mapwrite 0x00c3e6 thru 0x00e6f6 (0x2311 bytes) 000816 1790896879.076458 read 0x00abf9 thru 0x0103f5 (0x57fd bytes) 000817 1790896879.077180 mapread 0x033538 thru 0x0360ca (0x2b93 bytes) 000818 1790896879.078647 mapread 0x022654 thru 0x027118 (0x4ac5 bytes) 000819 1790896879.079401 read 0x00101b thru 0x0074a8 (0x648e bytes) 000823 1790896879.081689 mapread 0x004f38 thru 0x0067a8 (0x1871 bytes) 000825 1790896879.083196 punch from 0x030a36 to 0x03b81c (0xade7 bytes) 000827 1790896879.091044 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd63e64a8 (stale) Mirror 14: 0xd63e64a8 (stale) Mirror 15: 0xd63e64a8 (stale) Mirror 16: 0xd63e64a8 (stale) Mirror 19: 0xd63e64a8 (stale) Mirror 20: 0xd63e64a8 (stale) Mirror 22: 0xd63e64a8 (stale) Mirror 24: 0xd63e64a8 (stale) Mirror 25: 0xd63e64a8 (stale) Mirror 26: 0xd63e64a8 (stale) Mirror 27: 0xd63e64a8 (stale) Mirror 29: 0xd63e64a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd63e64a8 (stale) Mirror 14: 0xd63e64a8 (stale) Mirror 15: 0xd63e64a8 (stale) Mirror 16: 0xd63e64a8 (stale) Mirror 19: 0xd63e64a8 (stale) Mirror 20: 0xd63e64a8 (stale) Mirror 22: 0xd63e64a8 (stale) Mirror 24: 0xd63e64a8 (stale) Mirror 25: 0xd63e64a8 (stale) Mirror 26: 0xd63e64a8 (stale) Mirror 27: 0xd63e64a8 (stale) Mirror 29: 0xd63e64a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000829 1790896879.625429 punch from 0x031cd2 to 0x036873 (0x4ba2 bytes) 000830 1790896879.629812 mapwrite 0x036c9e thru 0x037ddc (0x113f bytes) 000831 1790896879.681229 trunc from 0x040000 to 0x029d75 (0x1628a bytes) 000832 1790896879.698520 mapread 0x0151de thru 0x01820a (0x302d bytes) 000833 1790896879.710341 mapwrite 0x020640 thru 0x02ba7d (0xb43e bytes) 000836 1790896879.805141 trunc from 0x02ba7e to 0x03d086 (0x11609 bytes) 000837 1790896879.834285 mapread 0x0106e0 thru 0x016277 (0x5b98 bytes) 000838 1790896879.836443 mapread 0x03373d thru 0x037588 (0x3e4c bytes) 000840 1790896879.852577 read 0x004323 thru 0x007083 (0x2d61 bytes) 000841 1790896879.860394 read 0x035717 thru 0x03d086 (0x7970 bytes) 000843 1790896879.870878 punch from 0x0011ef to 0x00304c (0x1e5e bytes) 000844 1790896879.876423 trunc from 0x03d087 to 0x02c361 (0x10d25 bytes) 000845 1790896879.902197 punch from 0x0028c2 to 0x009d67 (0x74a6 bytes) 000846 1790896879.908430 trunc from 0x02c362 to 0x0110e0 (0x1b281 bytes) 000847 1790896879.929282 trunc from 0x0110e1 to 0x01af5d (0x9e7d bytes) 000848 1790896879.953435 write 0x03dc34 thru 0x03ffff (0x23cc bytes) 000851 1790896879.957794 read 0x01ef4e thru 0x0295ce (0xa681 bytes) 000852 1790896879.970004 write 0x00b1cd thru 0x0180fc (0xcf30 bytes) 000854 1790896879.973976 write 0x0014e8 thru 0x0063d1 (0x4eea bytes) 000855 1790896879.975888 mapread 0x035904 thru 0x037908 (0x2005 bytes) 000857 1790896879.978292 trunc from 0x040000 to 0x03cb6c (0x3493 bytes) 000859 1790896880.009577 read 0x02a867 thru 0x038e73 (0xe60d bytes) 000860 1790896880.016092 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cb6d): Mirror 14: 0x833dde10 (stale) Mirror 15: 0x833dde10 (stale) Mirror 16: 0x833dde10 (stale) Mirror 19: 0x833dde10 (stale) Mirror 20: 0x833dde10 (stale) Mirror 22: 0x833dde10 (stale) Mirror 24: 0x833dde10 (stale) Mirror 25: 0x833dde10 (stale) Mirror 26: 0x833dde10 (stale) Mirror 27: 0x833dde10 (stale) Mirror 29: 0x833dde10 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb6d): Mirror 14: 0x833dde10 (stale) Mirror 15: 0x833dde10 (stale) Mirror 16: 0x833dde10 (stale) Mirror 19: 0x833dde10 (stale) Mirror 20: 0x833dde10 (stale) Mirror 22: 0x833dde10 (stale) Mirror 24: 0x833dde10 (stale) Mirror 25: 0x833dde10 (stale) Mirror 26: 0x833dde10 (stale) Mirror 27: 0x833dde10 (stale) Mirror 29: 0x833dde10 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb6d: skipped 000862 1790896880.617435 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3cb6d): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb6d): Mirror 14: 0x833dde10 (stale) Mirror 15: 0x833dde10 (stale) Mirror 16: 0x833dde10 (stale) Mirror 19: 0x833dde10 (stale) Mirror 20: 0x833dde10 (stale) Mirror 22: 0x833dde10 (stale) Mirror 24: 0x833dde10 (stale) Mirror 25: 0x833dde10 (stale) Mirror 26: 0x833dde10 (stale) Mirror 27: 0x833dde10 (stale) Mirror 29: 0x833dde10 (stale) CRC-32 checksum value for chunk [0, 0x3cb6d): Mirror 1: 0xacde0221 Mirror 30: 0xacde0221 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb6d: skipped 000863 1790896882.138611 read 0x00fc2f thru 0x0157fb (0x5bcd bytes) 000864 1790896882.159186 write 0x025d04 thru 0x030f17 (0xb214 bytes) 000865 1790896882.286159 write 0x02c545 thru 0x02fd12 (0x37ce bytes) 000866 1790896882.287148 write 0x01f4fc thru 0x02da9c (0xe5a1 bytes) 000867 1790896882.294820 trunc from 0x03cb6d to 0x03f503 (0x2997 bytes) 000868 1790896882.317433 punch from 0x038b9f to 0x03f503 (0x6965 bytes) 000869 1790896882.331569 trunc from 0x03f504 to 0x013c22 (0x2b8e1 bytes) 000870 1790896882.353883 write 0x027d49 thru 0x034f4d (0xd205 bytes) 000871 1790896882.357474 write 0x003108 thru 0x00e878 (0xb771 bytes) 000876 1790896882.387035 read 0x025d93 thru 0x031d6b (0xbfd9 bytes) 000878 1790896882.399179 trunc from 0x034f4e to 0x005fd2 (0x2ef7b bytes) 000879 1790896882.426240 trunc from 0x005fd3 to 0x00458c (0x1a46 bytes) 000880 1790896882.463253 read 0x002c12 thru 0x00458c (0x197b bytes) 000881 1790896882.463832 mapwrite 0x02b4c8 thru 0x033eb3 (0x89ec bytes) 000885 1790896882.566352 punch from 0x0120ac to 0x01c9e4 (0xa939 bytes) 000886 1790896882.571808 read 0x032f98 thru 0x033eb3 (0xf1c bytes) 000887 1790896882.572701 write 0x03613b thru 0x039dea (0x3cb0 bytes) 000888 1790896882.574194 trunc from 0x039deb to 0x02fc5e (0xa18c bytes) 000890 1790896882.596463 trunc from 0x02fc5f to 0x006ce0 (0x28f7e bytes) 000891 1790896882.620089 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6ce1): Mirror 14: 0xccc35775 (stale) Mirror 15: 0xccc35775 (stale) Mirror 16: 0xccc35775 (stale) Mirror 19: 0xccc35775 (stale) Mirror 20: 0xccc35775 (stale) Mirror 22: 0xccc35775 (stale) Mirror 24: 0xccc35775 (stale) Mirror 25: 0xccc35775 (stale) Mirror 26: 0xccc35775 (stale) Mirror 27: 0xccc35775 (stale) Mirror 29: 0xccc35775 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6ce1): Mirror 14: 0xccc35775 (stale) Mirror 15: 0xccc35775 (stale) Mirror 16: 0xccc35775 (stale) Mirror 19: 0xccc35775 (stale) Mirror 20: 0xccc35775 (stale) Mirror 22: 0xccc35775 (stale) Mirror 24: 0xccc35775 (stale) Mirror 25: 0xccc35775 (stale) Mirror 26: 0xccc35775 (stale) Mirror 27: 0xccc35775 (stale) Mirror 29: 0xccc35775 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6ce1: skipped 000894 1790896883.119150 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x6ce1): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6ce1): Mirror 14: 0xccc35775 (stale) Mirror 15: 0xccc35775 (stale) Mirror 16: 0xccc35775 (stale) Mirror 19: 0xccc35775 (stale) Mirror 20: 0xccc35775 (stale) Mirror 22: 0xccc35775 (stale) Mirror 24: 0xccc35775 (stale) Mirror 25: 0xccc35775 (stale) Mirror 26: 0xccc35775 (stale) Mirror 27: 0xccc35775 (stale) Mirror 29: 0xccc35775 (stale) CRC-32 checksum value for chunk [0, 0x6ce1): Mirror 1: 0xc14c4e43 Mirror 30: 0xc14c4e43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6ce1: skipped 000897 1790896884.233543 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x6ce1): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6ce1): Mirror 14: 0xccc35775 (stale) Mirror 15: 0xccc35775 (stale) Mirror 16: 0xccc35775 (stale) Mirror 19: 0xccc35775 (stale) Mirror 20: 0xccc35775 (stale) Mirror 22: 0xccc35775 (stale) Mirror 24: 0xccc35775 (stale) Mirror 25: 0xccc35775 (stale) Mirror 26: 0xccc35775 (stale) Mirror 27: 0xccc35775 (stale) Mirror 29: 0xccc35775 (stale) CRC-32 checksum value for chunk [0, 0x6ce1): Mirror 1: 0xc14c4e43 Mirror 30: 0xc14c4e43 Mirror 31: 0xc14c4e43 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6ce1: skipped 000899 1790896885.430835 write 0x03fca6 thru 0x03ffff (0x35a bytes) 000901 1790896885.559591 mapwrite 0x036690 thru 0x03c1ee (0x5b5f bytes) 000903 1790896885.680278 write 0x02ce57 thru 0x02fab9 (0x2c63 bytes) 000904 1790896885.691298 mapwrite 0x031ee0 thru 0x03fd99 (0xdeba bytes) 000905 1790896885.778980 read 0x02fd10 thru 0x038c74 (0x8f65 bytes) 000907 1790896885.786106 read 0x03a165 thru 0x03ffff (0x5e9b bytes) 000908 1790896885.786853 mapread 0x0298d7 thru 0x0367a5 (0xcecf bytes) 000909 1790896885.818280 trunc from 0x040000 to 0x004e95 (0x3b16a bytes) 000910 1790896885.846875 write 0x015961 thru 0x023d0d (0xe3ad bytes) 000911 1790896885.852051 read 0x019123 thru 0x023d0d (0xabeb bytes) 000912 1790896885.852854 mapread 0x009d1a thru 0x0195ad (0xf894 bytes) 000913 1790896885.863116 punch from 0x01c10b to 0x01e7a1 (0x2697 bytes) 000914 1790896885.875389 write 0x03297c thru 0x033ddd (0x1462 bytes) 000915 1790896885.877878 trunc from 0x033dde to 0x0309c7 (0x3416 bytes) 000916 1790896885.904257 mapwrite 0x03391e thru 0x03ffff (0xc6e2 bytes) 000919 1790896886.022007 punch from 0x00b797 to 0x01566b (0x9ed5 bytes) 000920 1790896886.028253 trunc from 0x040000 to 0x014f36 (0x2b0c9 bytes) 000922 1790896886.112068 mapread 0x002947 thru 0x005c33 (0x32ed bytes) 000925 1790896886.115434 write 0x03a22b thru 0x03ffff (0x5dd5 bytes) 000927 1790896886.117584 mapread 0x013b47 thru 0x0224bc (0xe976 bytes) 000928 1790896886.236551 read 0x02d4b7 thru 0x036e7d (0x99c7 bytes) 000929 1790896886.250192 trunc from 0x040000 to 0x00c8cf (0x33730 bytes) 000930 1790896886.277074 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0xc8d0): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc8d0): Mirror 14: 0x1478813c (stale) Mirror 15: 0x1478813c (stale) Mirror 16: 0x1478813c (stale) Mirror 19: 0x1478813c (stale) Mirror 20: 0x1478813c (stale) Mirror 22: 0x1478813c (stale) Mirror 24: 0x1478813c (stale) Mirror 25: 0x1478813c (stale) Mirror 26: 0x1478813c (stale) Mirror 27: 0x1478813c (stale) Mirror 29: 0x1478813c (stale) Mirror 30: 0x418f65e3 (stale) Mirror 31: 0x418f65e3 (stale) CRC-32 checksum value for chunk [0, 0xc8d0): Mirror 1: 0x418f65e3 Mirror 32: 0x418f65e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc8d0: skipped 000931 1790896887.342983 MIRROR_RESYNC 15 mirrors 000933 1790896889.970083 read 0x002b6b thru 0x003811 (0xca7 bytes) 000935 1790896889.975303 write 0x00d195 thru 0x0156e1 (0x854d bytes) 000937 1790896890.050374 mapread 0x0031e3 thru 0x013195 (0xffb3 bytes) 000938 1790896890.074210 read 0x0123a5 thru 0x0156e1 (0x333d bytes) 000939 1790896890.075327 trunc from 0x0156e2 to 0x022d48 (0xd667 bytes) 000940 1790896890.158530 mapwrite 0x023de7 thru 0x02636f (0x2589 bytes) 000941 1790896890.382628 mapread 0x01495b thru 0x01d4f1 (0x8b97 bytes) 000943 1790896890.384105 mapread 0x01ff91 thru 0x02636f (0x63df bytes) 000944 1790896890.389603 mapwrite 0x01a3ae thru 0x01b694 (0x12e7 bytes) 000946 1790896890.483842 read 0x00e9b5 thru 0x013cb5 (0x5301 bytes) 000948 1790896890.486967 trunc from 0x026370 to 0x038a29 (0x126ba bytes) 000949 1790896890.510034 read 0x026058 thru 0x026cb8 (0xc61 bytes) 000950 1790896890.510834 punch from 0x00e10c to 0x01af9a (0xce8f bytes) 000952 1790896890.528236 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x38a2a): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a2a): Mirror 14: 0x66d98f8 (stale) Mirror 15: 0x66d98f8 (stale) Mirror 16: 0x66d98f8 (stale) Mirror 19: 0x66d98f8 (stale) Mirror 20: 0x66d98f8 (stale) Mirror 22: 0x66d98f8 (stale) Mirror 24: 0x66d98f8 (stale) Mirror 25: 0x66d98f8 (stale) Mirror 26: 0x66d98f8 (stale) Mirror 27: 0x66d98f8 (stale) Mirror 29: 0x66d98f8 (stale) Mirror 30: 0x66d98f8 (stale) Mirror 31: 0x66d98f8 (stale) Mirror 32: 0x66d98f8 (stale) CRC-32 checksum value for chunk [0, 0x38a2a): Mirror 1: 0x5f851189 Mirror 33: 0x5f851189 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a2a: skipped 000953 1790896892.111533 trunc from 0x038a2a to 0x01fc08 (0x18e21 bytes) 000955 1790896892.254289 trunc from 0x01fc09 to 0x00bfd7 (0x13c31 bytes) 000956 1790896892.282649 mapwrite 0x00c51b thru 0x015e79 (0x995f bytes) 000957 1790896892.385822 trunc from 0x015e7a to 0x01e94a (0x8ad1 bytes) 000958 1790896892.401957 mapread 0x00f910 thru 0x017504 (0x7bf5 bytes) 000959 1790896892.412001 punch from 0x01718c to 0x01e94a (0x77bf bytes) 000962 1790896892.422233 write 0x02a073 thru 0x035bd2 (0xbb60 bytes) 000964 1790896892.427647 punch from 0x02eb50 to 0x035bd2 (0x7083 bytes) 000965 1790896892.443059 punch from 0x01f1ca to 0x021e9b (0x2cd2 bytes) 000966 1790896892.447754 trunc from 0x035bd3 to 0x0275e1 (0xe5f1 bytes) 000968 1790896892.482230 mapwrite 0x03550b thru 0x03d5c4 (0x80ba bytes) 000971 1790896892.549894 mapwrite 0x038b7f thru 0x03ffff (0x7481 bytes) 000972 1790896892.615654 MIRROR_SPLIT mirror 4 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x9d91876 (stale) Mirror 15: 0x9d91876 (stale) Mirror 16: 0x9d91876 (stale) Mirror 20: 0x9d91876 (stale) Mirror 22: 0x9d91876 (stale) Mirror 24: 0x9d91876 (stale) Mirror 25: 0x9d91876 (stale) Mirror 26: 0x9d91876 (stale) Mirror 27: 0x9d91876 (stale) Mirror 29: 0x9d91876 (stale) Mirror 30: 0x9d91876 (stale) Mirror 31: 0x9d91876 (stale) Mirror 32: 0x9d91876 (stale) Mirror 33: 0xd5943084 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x9d91876 (stale) Mirror 15: 0x9d91876 (stale) Mirror 16: 0x9d91876 (stale) Mirror 20: 0x9d91876 (stale) Mirror 22: 0x9d91876 (stale) Mirror 24: 0x9d91876 (stale) Mirror 25: 0x9d91876 (stale) Mirror 26: 0x9d91876 (stale) Mirror 27: 0x9d91876 (stale) Mirror 29: 0x9d91876 (stale) Mirror 30: 0x9d91876 (stale) Mirror 31: 0x9d91876 (stale) Mirror 32: 0x9d91876 (stale) Mirror 33: 0xd5943084 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000973 1790896893.347165 punch from 0x023871 to 0x02770f (0x3e9f bytes) 000975 1790896893.363439 write 0x025f56 thru 0x026243 (0x2ee bytes) 000976 1790896893.380185 trunc from 0x040000 to 0x03bf25 (0x40da bytes) 000979 1790896893.412457 MIRROR_RESYNC 15 mirrors 000980 1790896898.201364 MIRROR_EXTEND to 16 mirrors 000981 1790896899.100133 mapwrite 0x024bc2 thru 0x027656 (0x2a95 bytes) 000982 1790896899.284614 mapwrite 0x00a84c thru 0x010157 (0x590c bytes) 000983 1790896899.354884 mapread 0x029762 thru 0x038d8a (0xf629 bytes) 000984 1790896899.374109 read 0x020269 thru 0x028184 (0x7f1c bytes) 000985 1790896899.374917 read 0x00ac2c thru 0x0179b3 (0xcd88 bytes) 000986 1790896899.383864 punch from 0x034e53 to 0x037fcb (0x3179 bytes) 000987 1790896899.388141 punch from 0x01afe7 to 0x024522 (0x953c bytes) 000988 1790896899.392967 mapwrite 0x01a326 thru 0x022231 (0x7f0c bytes) 000989 1790896899.507868 punch from 0x02789a to 0x03022c (0x8993 bytes) 000991 1790896899.519787 punch from 0x031726 to 0x039c78 (0x8553 bytes) 000992 1790896899.535240 MIRROR_RESYNC 16 mirrors 000993 1790896902.811946 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20b:0x0]: Invalid argument (22) 000994 1790896903.523939 read 0x03a6c0 thru 0x03bf25 (0x1866 bytes) 000995 1790896903.529422 write 0x037376 thru 0x03a851 (0x34dc bytes) 000997 1790896903.630631 mapwrite 0x027786 thru 0x03719f (0xfa1a bytes) 000998 1790896903.738978 trunc from 0x03bf26 to 0x0096c5 (0x32860 bytes) 000999 1790896903.765832 mapread 0x007da9 thru 0x0096c5 (0x191d bytes) 001000 1790896903.779533 punch from 0x006e48 to 0x007168 (0x321 bytes) All operations completed A-OK! PASS 70a (137s) == sanity-flr test 71: check mirror extend parent fid ==== 19:21:49 (1790896909) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 19:21:53 (1790896913) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211576 s, 9.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137953 s, 7.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 19:22:01 (1790896921) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4247328 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 599456 .. done Extending file size to 8529920 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5581952 .. Extending file size to 8733312 .. Extending file size to 8543168 .. Extending file size to 8794272 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9025920 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9143776 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9258752 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 58073 58075 58076 58079 58081 failed 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) 192.168.204.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:23:25 (1790897005) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6227424 .. Extending file size to 1781184 .. Extending file size to 6367296 .. Extending file size to 7352128 .. Extending file size to 6889632 .. Extending file size to 7605408 .. Extending file size to 7974304 .. Extending file size to 8118016 .. Extending file size to 8070624 .. Extending file size to 8885600 .. Extending file size to 8308288 .. Extending file size to 8420256 .. Extending file size to 9122752 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9357472 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8705600 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 59627 59628 59630 59631 59633 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) 192.168.204.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:24:51 (1790897091) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.152048 s, 13.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.299355 s, 7.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xee967050 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:25:09 (1790897109) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0862852 s, 24.3 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 19:25:16 (1790897116) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x258:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:25:24 (1790897124) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.248477 s, 8.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:25:33 (1790897133) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.309163 s, 6.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:25:42 (1790897142) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.292811 s, 7.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:26:17 (1790897177) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130075 s, 8.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:27:20 (1790897240) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.245522 s, 8.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:28:25 (1790897305) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg438-server mds-ost sync done. Creating new pool test_204f oleg438-server: Pool lustre.test_204f created Adding targets to pool oleg438-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg438-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.439918 s, 4.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x310:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg438-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg438-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg438-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:29:48 (1790897388) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:29:56 (1790897396) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:30:05 (1790897405) oleg438-server: Pool lustre.test_206 created Adding targets to pool oleg438-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg438-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg438-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg438-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg438-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:30:29 (1790897429) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947708 s, 11.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092713 s, 11.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xaf325674 Mirror 3: 0xaf325674 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:30:42 (1790897442) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:30:46 (1790897446) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:30:50 (1790897450) Starting client: oleg438-client.virtnet: -o user_xattr,flock 192.168.204.138@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.15731 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xe90000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xe90000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xe90000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1280000 Read this (4 bytes): BBBB 192.168.204.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg438-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:31:29 (1790897489) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0141821 s, 578 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:31:38 (1790897498) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616419 s, 17.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x27d lmm_fid: [0x200000404:0x27d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x27d lmm_fid: [0x200000404:0x27d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x31a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834557 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-01 19:31:39.000000000 -0400 Modify: 2026-10-01 19:31:39.000000000 -0400 Change: 2026-10-01 19:31:42.000000000 -0400 Birth: 2026-10-01 19:31:40.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:31:48 (1790897508) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:31:51 (1790897511) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.821673 s, 49.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0844419 s, 48.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 1 19:31 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x27f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 1 19:31 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:32:03 (1790897523) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:32:14 (1790897534) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1981 sec =========== 19:32:27 (1790897547) === sanity-flr: start cleanup 19:32:29 (1790897549) === === sanity-flr: finish cleanup 19:32:37 (1790897557) ===