-----============= acceptance-small: sanity-flr ============----- Tue Sep 1 07:56:26 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg338-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:56:50 (1788263810) === oleg338-client.virtnet: executing check_config_client /mnt/lustre oleg338-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg338-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fe807d49800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fe807d49800.idle_timeout=debug disable quota as required oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:57:30 (1788263850) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8227] == sanity-flr test 0a: lfs mirror create with -N option == 07:57:33 (1788263853) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:57:52 (1788263872) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:57:55 (1788263875) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:57:58 (1788263878) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:58:20 (1788263900) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:58:22 (1788263902) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 07:58:25 (1788263905) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9fe807d49800.stats=clear osc.lustre-OST0001-osc-ffff9fe807d49800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:58:32 (1788263912) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:58:42 (1788263922) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:58:50 (1788263930) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:58:53 (1788263933) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:59:00 (1788263940) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 07:59:08 (1788263948) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:59:16 (1788263956) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:59:23 (1788263963) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:59:31 (1788263971) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:59:42 (1788263982) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:59:49 (1788263989) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.940225 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760032 s, 13.8 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:59:58 (1788263998) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.60974 s, 27.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:00:06 (1788264006) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:00:17 (1788264017) Stopping /mnt/lustre-ost1 (opts:) on oleg338-server oleg338-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fe807d49800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe807d49800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg338-server: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg338-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg338-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg338-server oleg338-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fe807d49800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe807d49800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg338-server: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg338-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (134s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:02:30 (1788264150) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.174766 s, 18.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg338-server oleg338-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fe807d49800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe807d49800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg338-server: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg338-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:03:12 (1788264192) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg338-server oleg338-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fe807d49800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe807d49800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg338-server: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg338-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg338-server oleg338-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fe807d49800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe807d49800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg338-server: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg338-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:04:40 (1788264280) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.05509 s, 17.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 1 08:04 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.940s user 0m0.000s sys 0m0.167s Stopping /mnt/lustre-ost1 (opts:) on oleg338-server oleg338-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fe807d49800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe807d49800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.906s user 0m0.001s sys 0m0.173s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg338-server: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg338-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg338-server oleg338-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fe807d49800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe807d49800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.104s user 0m0.000s sys 0m0.188s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg338-server: oleg338-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg338-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg338-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fe807d49800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe807d49800.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:06:10 (1788264370) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:06:15 (1788264375) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:06:19 (1788264379) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 08:06:23 (1788264383) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 08:06:30 (1788264390) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.570605 s, 22.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:06:43 (1788264403) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449637 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246449 s, 42.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:06:51 (1788264411) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:07:00 (1788264420) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00130003 s, 788 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00112813 s, 908 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0076305 s, 134 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0056844 s, 180 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 08:07:10 (1788264430) create test files with size 15755713 .. f462075d80ba8a3d4abf910ddd7d742b - 9d495c16a708e68b23536a31cbada994 - 4453f9f3fb434d9fed1365894f0107e1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg338-client.virtnet /mnt/lustre (opts:) Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 08:07:35 (1788264455) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:07:49 (1788264469) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:07:55 (1788264475) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788817 s, 13.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283006 s, 37.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xab:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xac:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 08:08:04 (1788264484) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0775199 s, 27.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.366005 s, 11.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135336 s, 15.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 08:08:19 (1788264499) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 08:08:22 (1788264502) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:08:26 (1788264506) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638576 s, 16.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 08:08:34 (1788264514) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.364767 s, 8.6 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:08:44 (1788264524) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:08:54 (1788264534) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.365811 s, 28.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg338-client.virtnet /mnt/lustre (opts:) Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:09:05 (1788264545) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9fe820ed7000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.460061 s, 22.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg338-client.virtnet /mnt/lustre (opts:) Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9fe809752000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:09:22 (1788264562) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:10:02 (1788264602) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:10:41 (1788264641) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 08:10:44 (1788264644) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.220937 s, 14.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.257223 s, 12.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1d097155 (nosync,stale) Mirror 2: 0x1d097155 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 05051cd6b5687fbead6072c411a3e304 - mirror 1 checksum: 7e84a405ef734529ceb6600ed789ea93 - mirror 2 checksum: 05051cd6b5687fbead6072c411a3e304 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-01T08:10:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-01T08:10:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 05051cd6b5687fbead6072c411a3e304 - mirror 1 checksum: 7e84a405ef734529ceb6600ed789ea93 - mirror 2 checksum: 7e84a405ef734529ceb6600ed789ea93 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-01T08:10:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-01T08:10:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:10:56 (1788264656) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681128 s, 15.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:11:03 (1788264663) Failing mds1 on oleg338-server Stopping /mnt/lustre-mds1 (opts:) on oleg338-server 08:11:15 (1788264675) shut down facet: mds1 facet_host: oleg338-server facet_failover_host: oleg338-server Failover mds1 to oleg338-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 08:11:50 (1788264710) targets are mounted 08:11:50 (1788264710) facet_failover done oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351514 s, 29.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:12:06 (1788264726) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0419308 s, 488 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0146643 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:12:16 (1788264736) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0394333 s, 519 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0064729 s, 3.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:12:26 (1788264746) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00784855 s, 2.1 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:12:34 (1788264754) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0594629 s, 344 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:12:42 (1788264762) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0117334 s, 349 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0343643 s, 119 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:12:50 (1788264770) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:13:11 (1788264791) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:13:38 (1788264818) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:14:03 (1788264843) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:14:28 (1788264868) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 08:14:38 (1788264878) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57587 mirror split pid 57588 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57587 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57588 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 08:15:45 (1788264945) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4947 skipping zero size read fallocating to largest ever: 0x33529 000002 1788264947.842239 fallocate 0x02f66a thru 0x033528 (0x3ebf bytes) 000003 1788264947.847736 trunc from 00000000 to 0x01f846 (0x1f847 bytes) 000004 1788264947.869224 trunc from 0x01f847 to 0x031c5d (0x12417 bytes) 000005 1788264947.891807 mapwrite 0x01a040 thru 0x021935 (0x78f6 bytes) 000006 1788264947.955179 mapread 0x01280a thru 0x01e55f (0xbd56 bytes) 000008 1788264947.981298 mapwrite 0x013917 thru 0x01ba7c (0x8166 bytes) 000009 1788264948.060746 punch from 0x01b4dd to 0x01e717 (0x323b bytes) 000010 1788264948.073018 mapread 0x025854 thru 0x02a63c (0x4de9 bytes) 000011 1788264948.101098 mapread 0x0067ee thru 0x013039 (0xc84c bytes) 000012 1788264948.110047 mapread 0x02e890 thru 0x031c5d (0x33ce bytes) fallocating to largest ever: 0x395b1 000013 1788264948.119331 fallocate 0x02b16c thru 0x0395b0 (0xe445 bytes) 000014 1788264948.127442 write 0x02819e thru 0x02c8ad (0x4710 bytes) 000015 1788264948.129480 write 0x00dde7 thru 0x014ca1 (0x6ebb bytes) 000016 1788264948.131469 write 0x017f0b thru 0x025e9a (0xdf90 bytes) 000017 1788264948.134202 fallocate 0x00ed83 thru 0x016907 (0x7b85 bytes) 000019 1788264948.141840 write 0x0100a3 thru 0x013cbf (0x3c1d bytes) 000020 1788264948.149053 write 0x03d5c6 thru 0x03d6f4 (0x12f bytes) 000022 1788264948.156364 read 0x01cda6 thru 0x0224cc (0x5727 bytes) 000023 1788264948.163817 read 0x02a547 thru 0x03471e (0xa1d8 bytes) 000025 1788264948.207058 read 0x02c25e thru 0x0323e6 (0x6189 bytes) 000026 1788264948.207592 punch from 0x0375a9 to 0x038e59 (0x18b1 bytes) 000027 1788264948.224282 mapwrite 0x01d4c2 thru 0x01d4cd (0xc bytes) 000029 1788264948.284356 write 0x021b51 thru 0x0298d9 (0x7d89 bytes) 000030 1788264948.290442 MIRROR_EXTEND to 2 mirrors 000031 1788264948.990889 mapread 0x017d2a thru 0x01df72 (0x6249 bytes) 000032 1788264949.049743 mapread 0x02eae4 thru 0x02f6d2 (0xbef bytes) 000033 1788264949.079342 punch from 0x02cd30 to 0x03c369 (0xf63a bytes) 000034 1788264949.122853 trunc from 0x010ab3 to 0x012a82 (0x1fd0 bytes) 000035 1788264949.127492 write 0x0163d0 thru 0x019605 (0x3236 bytes) 000036 1788264949.157728 fallocate 0x01dce1 thru 0x026b24 (0x8e44 bytes) 000037 1788264949.177354 read 0x01c366 thru 0x027775 (0xb410 bytes) 000038 1788264949.212313 punch from 0x0173f5 to 0x02544b (0xe057 bytes) 000039 1788264949.248845 mapwrite 0x022c2e thru 0x031a68 (0xee3b bytes) 000040 1788264949.345507 mapread 0x001a2c thru 0x007ba8 (0x617d bytes) 000042 1788264949.356307 write 0x027365 thru 0x02c67d (0x5319 bytes) 000043 1788264949.358172 trunc from 0x03d6f5 to 0x001e78 (0x3b87c bytes) 000044 1788264949.411393 write 0x018f3e thru 0x01c82b (0x38ee bytes) 000045 1788264949.416240 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) 000046 1788264949.591722 punch from 0x019795 to 0x01c82b (0x3097 bytes) 000047 1788264949.596548 fallocate 0x02e8e5 thru 0x03687f (0x7f9b bytes) 000048 1788264949.601304 trunc from 0x01bab8 to 0x01c82b (0xd74 bytes) 000049 1788264949.607712 read 0x00bf2a thru 0x0182ee (0xc3c5 bytes) 000050 1788264949.619384 mapwrite 0x01290b thru 0x019d42 (0x7438 bytes) 000051 1788264949.636226 trunc from 0x00abdb to 0x00f79b (0x4bc1 bytes) 000052 1788264949.640166 mapwrite 0x013b90 thru 0x01c1b0 (0x8621 bytes) 000053 1788264949.664109 write 0x02c335 thru 0x036315 (0x9fe1 bytes) 000054 1788264949.670381 trunc from 0x021abc to 0x02989d (0x7de2 bytes) 000055 1788264949.675367 punch from 0x029b1e to 0x036315 (0xc7f8 bytes) 000056 1788264949.684831 mapwrite 0x021f76 thru 0x030c5d (0xece8 bytes) 000057 1788264949.716915 mapwrite 0x021422 thru 0x030f21 (0xfb00 bytes) 000058 1788264949.734435 write 0x029dc4 thru 0x039d8f (0xffcc bytes) 000059 1788264949.742430 trunc from 0x023e6d to 0x02c239 (0x83cd bytes) 000061 1788264949.774080 punch from 0x013fbe to 0x022fbd (0xf000 bytes) 000062 1788264949.793164 punch from 0x02def7 to 0x039d8f (0xbe99 bytes) 000064 1788264949.805713 mapread 0x032c91 thru 0x039d8f (0x70ff bytes) 000065 1788264949.836422 trunc from 0x039d90 to 0x00e511 (0x2b87e bytes) 000066 1788264949.864453 write 0x01778f thru 0x017f61 (0x7d3 bytes) 000067 1788264949.865405 mapwrite 0x029910 thru 0x037ca0 (0xe391 bytes) 000069 1788264949.979169 write 0x00eb56 thru 0x00fa0d (0xeb8 bytes) 000070 1788264949.986904 write 0x00bccf thru 0x018372 (0xc6a4 bytes) 000072 1788264950.004513 punch from 0x02ba9d to 0x035cc9 (0xa22d bytes) 000073 1788264950.010178 mapread 0x003354 thru 0x009c62 (0x690f bytes) 000074 1788264950.025801 punch from 0x021cbc to 0x02390a (0x1c4f bytes) 000077 1788264950.042898 punch from 0x00f057 to 0x01b364 (0xc30e bytes) 000079 1788264950.068185 mapread 0x015ba5 thru 0x0188ad (0x2d09 bytes) 000080 1788264950.079113 read 0x009db4 thru 0x00a70d (0x95a bytes) 000081 1788264950.079668 trunc from 0x037ca1 to 0x039219 (0x1579 bytes) 000084 1788264950.098010 fallocate 0x037603 thru 0x037c7b (0x679 bytes) 000085 1788264950.120222 read 0x02123a thru 0x029076 (0x7e3d bytes) 000086 1788264950.139572 trunc from 0x03921a to 0x01c137 (0x1d0e2 bytes) 000087 1788264950.168920 trunc from 0x01c138 to 0x01809f (0x4098 bytes) 000088 1788264950.209720 mapwrite 0x019b76 thru 0x020e6a (0x72f5 bytes) 000089 1788264950.337905 write 0x003662 thru 0x0111bc (0xdb5b bytes) 000090 1788264950.350354 MIRROR_EXTEND to 2 mirrors 000091 1788264951.007810 mapread 0x01506d thru 0x01fe56 (0xadea bytes) fallocating to largest ever: 0x40000 000092 1788264951.065727 fallocate 0x03cf14 thru 0x03ffff (0x30ec bytes) 000093 1788264951.110724 fallocate 0x037581 thru 0x03ffff (0x8a7f bytes) 000094 1788264951.115717 trunc from 0x040000 to 0x03ac28 (0x53d7 bytes) 000095 1788264951.156572 read 0x00c7d6 thru 0x01b5e2 (0xee0d bytes) 000096 1788264951.167079 mapwrite 0x008aa2 thru 0x00e1e2 (0x5741 bytes) 000098 1788264951.190811 punch from 0x008910 to 0x00b826 (0x2f17 bytes) 000099 1788264951.195284 mapwrite 0x02dc78 thru 0x03908d (0xb416 bytes) 000100 1788264951.234449 mapwrite 0x01b141 thru 0x02a181 (0xf041 bytes) 000102 1788264951.267639 MIRROR_RESYNC 2 mirrors 000103 1788264951.934143 mapread 0x022ed2 thru 0x03033c (0xd46b bytes) 000104 1788264951.956688 write 0x035798 thru 0x03a653 (0x4ebc bytes) 000105 1788264951.994462 read 0x025151 thru 0x02db56 (0x8a06 bytes) 000106 1788264952.009092 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) 000107 1788264952.154322 trunc from 0x03ac29 to 0x00121f (0x39a09 bytes) 000108 1788264952.220991 trunc from 0x0010ac to 0x00121f (0x174 bytes) 000109 1788264952.226940 write 0x0176ff thru 0x01caba (0x53bc bytes) 000110 1788264952.228813 mapread 0x00f10b thru 0x01a4a1 (0xb397 bytes) 000111 1788264952.246259 punch from 0x00078d to 0x0040b4 (0x3928 bytes) 000112 1788264952.254870 MIRROR_EXTEND to 2 mirrors 000113 1788264952.854326 trunc from 0x01bc03 to 0x01caba (0xeb8 bytes) 000114 1788264952.902434 punch from 0x01a2c0 to 0x01caba (0x27fb bytes) 000115 1788264952.916742 fallocate 0x00a4e6 thru 0x00c471 (0x1f8c bytes) 000117 1788264952.955179 trunc from 0x01cabb to 0x02e2bb (0x11801 bytes) 000118 1788264953.069105 write 0x02f1cc thru 0x03a86d (0xb6a2 bytes) 000119 1788264953.076063 read 0x0107ce thru 0x01fda8 (0xf5db bytes) 000120 1788264953.110068 fallocate 0x0323eb thru 0x03ffff (0xdc15 bytes) 000121 1788264953.117139 write 0x004bbd thru 0x0141b1 (0xf5f5 bytes) 000122 1788264953.147387 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) 000123 1788264953.455403 trunc from 0x03a86e to 0x03ffed (0x5780 bytes) 000124 1788264953.493627 fallocate 0x017d88 thru 0x02488c (0xcb05 bytes) 000125 1788264953.505628 fallocate 0x027fe3 thru 0x02ee92 (0x6eb0 bytes) 000126 1788264953.529138 read 0x029cd6 thru 0x039041 (0xf36c bytes) 000127 1788264953.546820 punch from 0x01f938 to 0x02be7e (0xc547 bytes) 000128 1788264953.561210 fallocate 0x03fe9f thru 0x03ffff (0x161 bytes) 000130 1788264953.592543 fallocate 0x035de8 thru 0x03ffff (0xa218 bytes) 000131 1788264953.616742 read 0x01cdae thru 0x02468c (0x78df bytes) 000132 1788264953.667566 punch from 0x011183 to 0x01a1f3 (0x9071 bytes) 000133 1788264953.675230 mapread 0x006fc5 thru 0x0110a2 (0xa0de bytes) 000134 1788264953.681418 read 0x03f30e thru 0x03f9aa (0x69d bytes) 000135 1788264953.686077 trunc from 0x040000 to 0x028d1d (0x172e2 bytes) 000136 1788264953.706146 mapwrite 0x00993b thru 0x015b62 (0xc228 bytes) 000138 1788264953.771623 punch from 0x01378a to 0x01e8ed (0xb164 bytes) 000139 1788264953.785857 fallocate 0x0265ce thru 0x02c209 (0x5c3c bytes) 000140 1788264953.803669 MIRROR_EXTEND to 2 mirrors 000141 1788264954.993759 write 0x022294 thru 0x025d40 (0x3aad bytes) 000142 1788264955.034121 trunc from 0x028d1e to 0x035061 (0xc344 bytes) 000143 1788264955.053530 MIRROR_RESYNC 2 mirrors 000144 1788264955.908621 read 0x014004 thru 0x016abe (0x2abb bytes) 000145 1788264955.915477 trunc from 0x035062 to 0x019916 (0x1b74b bytes) 000146 1788264955.963747 read 0x00d178 thru 0x00e08e (0xf17 bytes) 000147 1788264955.978982 mapread 0x0090e5 thru 0x00f50e (0x642a bytes) 000148 1788264955.980010 trunc from 0x019917 to 0x015f6f (0x39a7 bytes) 000149 1788264956.016848 fallocate 0x0173fb thru 0x022b6d (0xb773 bytes) 000150 1788264956.030866 trunc from 0x022b6e to 0x03cfdb (0x1a46e bytes) 000151 1788264956.052437 fallocate 0x0269cc thru 0x027d4a (0x137f bytes) 000153 1788264956.058965 fallocate 0x009662 thru 0x009f23 (0x8c2 bytes) 000154 1788264956.079383 MIRROR_RESYNC 2 mirrors 000155 1788264956.759043 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) 000156 1788264956.924691 mapread 0x02874f thru 0x02abaa (0x245c bytes) 000157 1788264956.944400 trunc from 0x02bb69 to 0x0356dc (0x9b74 bytes) 000158 1788264956.959984 fallocate 0x00d46b thru 0x01bc0a (0xe7a0 bytes) 000159 1788264956.965794 read 0x00997e thru 0x01827b (0xe8fe bytes) 000160 1788264956.979499 trunc from 0x03cfdc to 0x00f0ce (0x2df0d bytes) 000161 1788264956.999998 read 0x000b80 thru 0x001ece (0x134f bytes) 000162 1788264957.008950 mapread 0x003c61 thru 0x00cd8c (0x912c bytes) 000164 1788264957.012738 mapwrite 0x03de7c thru 0x03ffff (0x2184 bytes) 000167 1788264957.098913 fallocate 0x00262b thru 0x00f083 (0xca59 bytes) 000168 1788264957.111087 write 0x00e87c thru 0x01aca4 (0xc429 bytes) 000170 1788264957.114359 trunc from 0x0031c8 to 0x00717f (0x3fb8 bytes) 000171 1788264957.119098 trunc from 0x03d7e7 to 0x03ffff (0x2819 bytes) 000172 1788264957.127862 fallocate 0x03282d thru 0x037d69 (0x553d bytes) 000174 1788264957.156086 fallocate 0x005340 thru 0x00c80c (0x74cd bytes) 000175 1788264957.175534 trunc from 0x0145ab to 0x01daa9 (0x94ff bytes) 000176 1788264957.214413 mapwrite 0x0229e5 thru 0x031147 (0xe763 bytes) 000177 1788264957.248681 fallocate 0x008bca thru 0x010876 (0x7cad bytes) 000178 1788264957.253260 write 0x00b764 thru 0x01780c (0xc0a9 bytes) 000179 1788264957.257672 trunc from 0x00be60 to 0x012c60 (0x6e01 bytes) 000180 1788264957.273228 trunc from 0x03bda8 to 0x03ffff (0x4258 bytes) 000181 1788264957.289666 read 0x01bb38 thru 0x021d9a (0x6263 bytes) 000182 1788264957.300971 punch from 0x00685a to 0x0160cc (0xf873 bytes) 000183 1788264957.306760 mapwrite 0x01c592 thru 0x021ac2 (0x5531 bytes) 000184 1788264957.322669 read 0x00cc51 thru 0x0167f1 (0x9ba1 bytes) 000185 1788264957.333405 trunc from 0x01e8e8 to 0x02a558 (0xbc71 bytes) 000186 1788264957.338051 mapread 0x0018d5 thru 0x00e7cf (0xcefb bytes) 000187 1788264957.339025 trunc from 0x040000 to 0x017ea3 (0x2815c bytes) 000188 1788264957.357518 mapwrite 0x010c35 thru 0x01ddb5 (0xd181 bytes) 000190 1788264957.518133 trunc from 0x01ddb6 to 0x02a220 (0xc46b bytes) 000191 1788264957.548503 trunc from 0x00f999 to 0x01439d (0x4a05 bytes) 000192 1788264957.573866 mapwrite 0x030df0 thru 0x03ffff (0xf210 bytes) 000193 1788264957.919091 punch from 0x010ca0 to 0x01cf01 (0xc262 bytes) 000195 1788264957.932178 trunc from 0x011b7a to 0x016ddc (0x5263 bytes) 000196 1788264957.945639 fallocate 0x02a000 thru 0x02e8ec (0x48ed bytes) 000197 1788264957.958048 read 0x0221f0 thru 0x02ff4f (0xdd60 bytes) 000198 1788264957.992349 punch from 0x003aa4 to 0x00eaf8 (0xb055 bytes) 000199 1788264958.001000 read 0x01159f thru 0x016ba6 (0x5608 bytes) 000200 1788264958.002304 mapread 0x020b8e thru 0x02ece7 (0xe15a bytes) 000201 1788264958.012217 mapread 0x02a5bc thru 0x0398ef (0xf334 bytes) 000202 1788264958.013162 mapwrite 0x0297fb thru 0x0372e3 (0xdae9 bytes) 000203 1788264958.091428 write 0x00e72a thru 0x017253 (0x8b2a bytes) 000204 1788264958.113539 fallocate 0x00bb54 thru 0x0142ab (0x8758 bytes) 000205 1788264958.120350 punch from 0x017164 to 0x01e0fd (0x6f9a bytes) 000206 1788264958.140034 mapwrite 0x01efb5 thru 0x023136 (0x4182 bytes) 000208 1788264958.172733 read 0x017ff6 thru 0x02245f (0xa46a bytes) 000209 1788264958.188564 trunc from 0x040000 to 0x036b37 (0x94c8 bytes) 000210 1788264958.209138 mapwrite 0x03f430 thru 0x03ffff (0xbd0 bytes) 000211 1788264958.263834 punch from 0x00cac7 to 0x012407 (0x5941 bytes) 000212 1788264958.280261 fallocate 0x031231 thru 0x031d39 (0xb09 bytes) 000213 1788264958.286060 read 0x039fd4 thru 0x03dbdf (0x3c0c bytes) 000215 1788264958.301075 fallocate 0x0219fb thru 0x0280f9 (0x66ff bytes) 000216 1788264958.311330 MIRROR_EXTEND to 2 mirrors 000217 1788264959.182693 write 0x00f5a9 thru 0x012c15 (0x366d bytes) 000218 1788264959.246915 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcec292ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcf0e5141 Mirror 3: 0xcf0e5141 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000219 1788264960.400690 read 0x02ae1b thru 0x02debc (0x30a2 bytes) 000220 1788264960.421461 mapread 0x03dc44 thru 0x03ffff (0x23bc bytes) 000221 1788264960.466463 trunc from 0x040000 to 0x024955 (0x1b6aa bytes) 000222 1788264960.538189 mapwrite 0x00be78 thru 0x00d16e (0x12f7 bytes) 000223 1788264960.568400 punch from 0x00ae3c to 0x01194c (0x6b11 bytes) 000224 1788264960.575345 fallocate 0x011c6c thru 0x020e8b (0xf220 bytes) 000225 1788264960.582940 mapread 0x00bded thru 0x00c7fc (0xa10 bytes) 000226 1788264960.604302 mapread 0x006613 thru 0x015726 (0xf114 bytes) 000227 1788264960.620715 fallocate 0x0203ea thru 0x02930c (0x8f23 bytes) 000228 1788264960.637322 trunc from 0x02930d to 0x0193f5 (0xff17 bytes) 000229 1788264960.661596 fallocate 0x02ba88 thru 0x030c15 (0x518e bytes) 000230 1788264960.666378 fallocate 0x010f80 thru 0x01eab6 (0xdb37 bytes) 000231 1788264960.674481 trunc from 0x009b2f to 0x00e3a7 (0x4879 bytes) 000232 1788264960.688678 read 0x007204 thru 0x008d93 (0x1b90 bytes) 000233 1788264960.690413 mapwrite 0x01f5d9 thru 0x02870c (0x9134 bytes) 000234 1788264960.805237 MIRROR_RESYNC 3 mirrors 000235 1788264961.988037 mapwrite 0x01892b thru 0x01ab94 (0x226a bytes) 000236 1788264962.117081 punch from 0x0281c1 to 0x02870c (0x54c bytes) 000237 1788264962.193043 MIRROR_RESYNC 3 mirrors 000238 1788264962.960410 read 0x02481d thru 0x02870c (0x3ef0 bytes) 000239 1788264962.967413 trunc from 0x02870d to 0x023702 (0x500a bytes) 000240 1788264963.038464 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x23703): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23703): Mirror 2: 0xb8e263ac (stale) Mirror 3: 0xb8e263ac (stale) CRC-32 checksum value for chunk [0, 0x23703): Mirror 1: 0xb8e263ac Mirror 4: 0xb8e263ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23703: skipped 000241 1788264964.218030 trunc from 0x023703 to 0x00fa45 (0x13cbd bytes) 000242 1788264964.261235 read 0x00ca8f thru 0x00fa45 (0x2fb7 bytes) 000243 1788264964.266985 fallocate 0x03ca25 thru 0x03ffff (0x35db bytes) 000244 1788264964.272212 trunc from 0x040000 to 0x009bad (0x36452 bytes) 000246 1788264964.288727 trunc from 0x009bae to 0x03ee5b (0x352ae bytes) 000247 1788264964.303113 mapwrite 0x00601a thru 0x00acc1 (0x4ca8 bytes) 000248 1788264964.331326 trunc from 0x03ee5c to 0x027d84 (0x170d7 bytes) 000249 1788264964.366835 trunc from 0x027d85 to 0x02d258 (0x54d4 bytes) 000250 1788264964.396948 trunc from 0x018ffc to 0x022bc6 (0x9bcb bytes) 000251 1788264964.404855 mapwrite 0x01e32e thru 0x022cc6 (0x4999 bytes) 000252 1788264964.445416 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2d259): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d259): Mirror 2: 0x674bd005 (stale) Mirror 3: 0x674bd005 (stale) Mirror 4: 0x3cf88001 (stale) CRC-32 checksum value for chunk [0, 0x2d259): Mirror 1: 0x545cfefc Mirror 5: 0x545cfefc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d259: skipped 000253 1788264965.350061 write 0x00480b thru 0x009a84 (0x527a bytes) 000254 1788264965.396761 trunc from 0x00a250 to 0x01998e (0xf73f bytes) 000255 1788264965.402588 trunc from 0x015bd4 to 0x022d99 (0xd1c6 bytes) 000256 1788264965.411352 mapread 0x01e95d thru 0x02abd0 (0xc274 bytes) 000257 1788264965.447222 trunc from 0x02d259 to 0x008a64 (0x247f4 bytes) 000258 1788264965.542257 read 0x002156 thru 0x008a64 (0x690f bytes) 000259 1788264965.554409 trunc from 0x008a65 to 0x0212b3 (0x1884f bytes) 000260 1788264965.604813 punch from 0x00ab87 to 0x01a486 (0xf900 bytes) 000261 1788264965.623488 mapwrite 0x032958 thru 0x03ce38 (0xa4e1 bytes) 000262 1788264965.732710 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ce39): Mirror 3: 0x2e52f3a8 (stale) Mirror 4: 0xcb23aa46 (stale) Mirror 5: 0xbee0fc54 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce39): Mirror 3: 0x2e52f3a8 (stale) Mirror 4: 0xcb23aa46 (stale) Mirror 5: 0xbee0fc54 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce39: skipped 000263 1788264966.305989 trunc from 0x03ce39 to 0x000c02 (0x3c236 bytes) 000264 1788264966.351953 punch from 0x00058d to 0x000c02 (0x676 bytes) 000265 1788264966.372086 write 0x028d2b thru 0x037ced (0xefc3 bytes) 000266 1788264966.374938 write 0x0355a6 thru 0x03ffff (0xaa5a bytes) 000267 1788264966.379072 fallocate 0x00ede8 thru 0x014dc5 (0x5fde bytes) 000268 1788264966.388887 punch from 0x026249 to 0x03431e (0xe0d6 bytes) 000269 1788264966.422086 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe28438de (stale) Mirror 4: 0x13178f4a (stale) Mirror 5: 0xea4317b1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xac0ad399 Mirror 6: 0xac0ad399 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000270 1788264967.802401 trunc from 0x03222a to 0x03ffff (0xddd6 bytes) 000271 1788264967.855537 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe28438de (stale) Mirror 4: 0x13178f4a (stale) Mirror 5: 0xea4317b1 (stale) Mirror 6: 0xac0ad399 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 7: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000272 1788264968.937318 mapread 0x02da56 thru 0x03bc81 (0xe22c bytes) 000273 1788264968.979552 write 0x02df2e thru 0x02eeb5 (0xf88 bytes) 000274 1788264969.023175 trunc from 0x040000 to 0x02539d (0x1ac62 bytes) 000275 1788264969.043975 trunc from 0x00734d to 0x0163ad (0xf061 bytes) 000276 1788264969.057055 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2539e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2539e): Mirror 3: 0xc49205aa (stale) Mirror 4: 0x2467aba9 (stale) Mirror 5: 0x68ae9aee (stale) Mirror 6: 0x97d9125b (stale) Mirror 7: 0x97d9125b (stale) CRC-32 checksum value for chunk [0, 0x2539e): Mirror 1: 0x97d9125b Mirror 8: 0x97d9125b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2539e: skipped 000277 1788264970.197140 mapwrite 0x03c12d thru 0x03c326 (0x1fa bytes) 000281 1788264970.298439 punch from 0x00b6ec to 0x016c62 (0xb577 bytes) 000282 1788264970.305978 read 0x0314b1 thru 0x03b5a3 (0xa0f3 bytes) 000283 1788264970.317555 trunc from 0x01df11 to 0x023bba (0x5caa bytes) 000285 1788264970.328631 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c327): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c327): Mirror 3: 0x7316610d (stale) Mirror 4: 0xfd3c8465 (stale) Mirror 5: 0x22dfb38e (stale) Mirror 6: 0xf968516a (stale) Mirror 7: 0x10b54a42 (stale) Mirror 8: 0x10b54a42 (stale) CRC-32 checksum value for chunk [0, 0x3c327): Mirror 1: 0xeb5ed866 Mirror 9: 0xeb5ed866 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c327: skipped 000286 1788264971.275232 punch from 0x03661c to 0x03c326 (0x5d0b bytes) 000288 1788264971.327399 write 0x01c41d thru 0x029295 (0xce79 bytes) 000289 1788264971.357450 write 0x00b3c8 thru 0x00f635 (0x426e bytes) 000291 1788264971.378473 punch from 0x02ef80 to 0x032d92 (0x3e13 bytes) 000292 1788264971.384685 trunc from 0x03c327 to 0x03aa25 (0x1901 bytes) 000293 1788264971.433711 trunc from 0x03aa26 to 0x01a834 (0x201f1 bytes) 000294 1788264971.470118 punch from 0x0136e2 to 0x013e81 (0x7a0 bytes) 000295 1788264971.479124 read 0x01509b thru 0x01a834 (0x579a bytes) 000296 1788264971.494377 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a835): Mirror 4: 0x291e9962 (stale) Mirror 5: 0xcba94956 (stale) Mirror 6: 0x818f764d (stale) Mirror 7: 0x818f764d (stale) Mirror 8: 0x818f764d (stale) Mirror 9: 0x818f764d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a835): Mirror 4: 0x291e9962 (stale) Mirror 5: 0xcba94956 (stale) Mirror 6: 0x818f764d (stale) Mirror 7: 0x818f764d (stale) Mirror 8: 0x818f764d (stale) Mirror 9: 0x818f764d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a835: skipped 000297 1788264972.131453 fallocate 0x039ac6 thru 0x03ffff (0x653a bytes) 000298 1788264972.144699 punch from 0x014881 to 0x0194d0 (0x4c50 bytes) 000300 1788264972.153644 write 0x03dada thru 0x03ffff (0x2526 bytes) 000301 1788264972.172648 punch from 0x038ab4 to 0x03ffff (0x754c bytes) 000302 1788264972.193494 read 0x009459 thru 0x00fbe8 (0x6790 bytes) 000303 1788264972.202220 trunc from 0x040000 to 0x032d04 (0xd2fb bytes) 000305 1788264972.240482 mapwrite 0x029b77 thru 0x03172a (0x7bb4 bytes) 000306 1788264972.295191 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32d05): Mirror 4: 0xb48c8719 (stale) Mirror 6: 0x5b3af758 (stale) Mirror 7: 0x5b3af758 (stale) Mirror 8: 0x5b3af758 (stale) Mirror 9: 0x5b3af758 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d05): Mirror 4: 0xb48c8719 (stale) Mirror 6: 0x5b3af758 (stale) Mirror 7: 0x5b3af758 (stale) Mirror 8: 0x5b3af758 (stale) Mirror 9: 0x5b3af758 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d05: skipped 000307 1788264972.759865 mapwrite 0x03501c thru 0x03ffff (0xafe4 bytes) 000308 1788264972.848691 punch from 0x03dc7a to 0x03ffff (0x2386 bytes) 000310 1788264972.861547 read 0x0246db thru 0x02b6d7 (0x6ffd bytes) 000311 1788264972.873024 write 0x01ad23 thru 0x02515b (0xa439 bytes) 000312 1788264972.878209 read 0x0265cc thru 0x030725 (0xa15a bytes) 000313 1788264972.887277 read 0x022741 thru 0x025a70 (0x3330 bytes) 000314 1788264972.888206 fallocate 0x02ebec thru 0x030733 (0x1b48 bytes) 000315 1788264972.892641 trunc from 0x025825 to 0x032dfc (0xd5d8 bytes) 000316 1788264972.911157 mapwrite 0x008215 thru 0x0089b3 (0x79f bytes) 000317 1788264972.931027 mapwrite 0x031ff9 thru 0x03225a (0x262 bytes) 000318 1788264972.955462 trunc from 0x040000 to 0x03b634 (0x49cb bytes) 000320 1788264972.975348 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3b635): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b635): Mirror 4: 0x209f184b (stale) Mirror 6: 0x67639483 (stale) Mirror 7: 0x289c55c1 (stale) Mirror 8: 0x289c55c1 (stale) Mirror 9: 0x289c55c1 (stale) CRC-32 checksum value for chunk [0, 0x3b635): Mirror 1: 0xfba4a5c8 Mirror 10: 0xfba4a5c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b635: skipped 000322 1788264974.037127 fallocate 0x0194d1 thru 0x01d787 (0x42b7 bytes) 000323 1788264974.085513 fallocate 0x01521c thru 0x022d18 (0xdafd bytes) 000324 1788264974.090552 punch from 0x02615b to 0x034fc6 (0xee6c bytes) 000325 1788264974.101142 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b635): Mirror 6: 0x67639483 (stale) Mirror 7: 0x289c55c1 (stale) Mirror 8: 0x289c55c1 (stale) Mirror 9: 0x289c55c1 (stale) Mirror 10: 0xfba4a5c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b635): Mirror 6: 0x67639483 (stale) Mirror 7: 0x289c55c1 (stale) Mirror 8: 0x289c55c1 (stale) Mirror 9: 0x289c55c1 (stale) Mirror 10: 0xfba4a5c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b635: skipped 000326 1788264974.640429 mapread 0x02e95d thru 0x02f001 (0x6a5 bytes) 000327 1788264974.654055 read 0x00bf91 thru 0x0150dc (0x914c bytes) 000328 1788264974.661240 mapread 0x021130 thru 0x023cfb (0x2bcc bytes) 000329 1788264974.669327 punch from 0x011fd9 to 0x01ad85 (0x8dad bytes) 000330 1788264974.673506 trunc from 0x022918 to 0x032250 (0xf939 bytes) 000331 1788264974.681475 trunc from 0x01c4d1 to 0x01d26f (0xd9f bytes) 000332 1788264974.686656 mapwrite 0x01fe76 thru 0x02165d (0x17e8 bytes) 000333 1788264974.710649 mapwrite 0x0243b4 thru 0x031682 (0xd2cf bytes) 000334 1788264974.787325 mapread 0x027084 thru 0x0340b3 (0xd030 bytes) 000335 1788264974.789282 write 0x027886 thru 0x02ffc7 (0x8742 bytes) 000336 1788264974.794083 read 0x0156d8 thru 0x021d28 (0xc651 bytes) 000337 1788264974.821453 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b635): Mirror 7: 0x289c55c1 (stale) Mirror 8: 0x289c55c1 (stale) Mirror 9: 0x289c55c1 (stale) Mirror 10: 0xfba4a5c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b635): Mirror 7: 0x289c55c1 (stale) Mirror 8: 0x289c55c1 (stale) Mirror 9: 0x289c55c1 (stale) Mirror 10: 0xfba4a5c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b635: skipped 000339 1788264975.195341 write 0x000158 thru 0x007314 (0x71bd bytes) 000340 1788264975.213232 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3b635): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b635): Mirror 7: 0x289c55c1 (stale) Mirror 8: 0x289c55c1 (stale) Mirror 9: 0x289c55c1 (stale) Mirror 10: 0xfba4a5c8 (stale) CRC-32 checksum value for chunk [0, 0x3b635): Mirror 1: 0xf1caec80 Mirror 11: 0xf1caec80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b635: skipped 000342 1788264976.466268 punch from 0x01441f to 0x01fa53 (0xb635 bytes) 000343 1788264976.520606 trunc from 0x0135c5 to 0x01e105 (0xab41 bytes) 000344 1788264976.525753 trunc from 0x01ceff to 0x02aeb5 (0xdfb7 bytes) 000345 1788264976.530741 punch from 0x02ecaa to 0x03161f (0x2976 bytes) 000346 1788264976.536573 mapread 0x00f771 thru 0x01d762 (0xdff2 bytes) 000347 1788264976.569941 fallocate 0x02c8bb thru 0x036907 (0xa04d bytes) 000348 1788264976.575393 trunc from 0x023d55 to 0x030d18 (0xcfc4 bytes) 000349 1788264976.584858 punch from 0x0231e0 to 0x029eba (0x6cdb bytes) 000350 1788264976.591188 write 0x03c89c thru 0x03ffff (0x3764 bytes) 000351 1788264976.596589 fallocate 0x00c969 thru 0x0117d1 (0x4e69 bytes) 000352 1788264976.606843 trunc from 0x040000 to 0x021a08 (0x1e5f7 bytes) 000353 1788264976.640412 trunc from 0x021a09 to 0x019f5f (0x7aa9 bytes) 000354 1788264976.664205 MIRROR_RESYNC 6 mirrors 000355 1788264977.926265 punch from 0x00d3ff to 0x014eaa (0x7aac bytes) 000356 1788264977.971153 punch from 0x000d09 to 0x00a65e (0x9956 bytes) 000357 1788264977.975454 mapwrite 0x01f729 thru 0x026889 (0x7161 bytes) 000358 1788264978.059229 fallocate 0x0198e1 thru 0x0213c5 (0x7ae5 bytes) 000359 1788264978.069389 trunc from 0x00e0e1 to 0x014410 (0x6330 bytes) 000360 1788264978.091245 punch from 0x006574 to 0x011270 (0xacfd bytes) 000361 1788264978.113925 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2688a): Mirror 7: 0x1003795b (stale) Mirror 9: 0x1003795b (stale) Mirror 10: 0x1003795b (stale) Mirror 11: 0x1003795b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2688a): Mirror 7: 0x1003795b (stale) Mirror 9: 0x1003795b (stale) Mirror 10: 0x1003795b (stale) Mirror 11: 0x1003795b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2688a: skipped 000362 1788264978.681990 punch from 0x00e89a to 0x01da28 (0xf18f bytes) 000363 1788264978.696897 write 0x00cebb thru 0x011106 (0x424c bytes) 000364 1788264978.754334 trunc from 0x001b76 to 0x01105b (0xf4e6 bytes) 000365 1788264978.783275 trunc from 0x02688a to 0x00519e (0x216eb bytes) 000366 1788264978.852293 mapread 0x001a52 thru 0x00519e (0x374d bytes) 000367 1788264978.876361 punch from 0x0044fe to 0x00519e (0xca1 bytes) 000369 1788264978.892309 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x519f): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x519f): Mirror 7: 0xd80137ad (stale) Mirror 9: 0xd80137ad (stale) Mirror 10: 0xd80137ad (stale) Mirror 11: 0xd80137ad (stale) CRC-32 checksum value for chunk [0, 0x519f): Mirror 1: 0x2acfa40c Mirror 12: 0x2acfa40c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x519f: skipped 000370 1788264980.021570 trunc from 0x003cac to 0x00519e (0x14f3 bytes) 000371 1788264980.066276 mapread 0x003666 thru 0x00519e (0x1b39 bytes) 000372 1788264980.085334 mapread 0x0037f4 thru 0x00519e (0x19ab bytes) 000373 1788264980.086476 punch from 0x001b1b to 0x004c6e (0x3154 bytes) 000374 1788264980.092514 trunc from 0x003915 to 0x00519e (0x188a bytes) 000375 1788264980.096948 trunc from 0x00519f to 0x03e4b8 (0x3931a bytes) 000376 1788264980.122685 punch from 0x00781a to 0x00a50b (0x2cf2 bytes) 000377 1788264980.131543 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e4b9): Mirror 9: 0x57211f34 (stale) Mirror 10: 0x57211f34 (stale) Mirror 11: 0x57211f34 (stale) Mirror 12: 0xe6ace1fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4b9): Mirror 9: 0x57211f34 (stale) Mirror 10: 0x57211f34 (stale) Mirror 11: 0x57211f34 (stale) Mirror 12: 0xe6ace1fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4b9: skipped 000379 1788264980.521959 read 0x016199 thru 0x01c642 (0x64aa bytes) 000380 1788264980.544691 fallocate 0x0232c9 thru 0x024b6d (0x18a5 bytes) 000381 1788264980.551035 mapwrite 0x00f7df thru 0x014e34 (0x5656 bytes) 000382 1788264980.590276 read 0x03c6a9 thru 0x03e4b8 (0x1e10 bytes) 000383 1788264980.622315 punch from 0x0075d0 to 0x009dd2 (0x2803 bytes) 000384 1788264980.632698 write 0x00c089 thru 0x00e9d6 (0x294e bytes) 000385 1788264980.639256 trunc from 0x00d540 to 0x00eae8 (0x15a9 bytes) 000386 1788264980.653427 trunc from 0x03e4b9 to 0x0003a2 (0x3e116 bytes) 000387 1788264980.676283 read 0x000115 thru 0x0003a2 (0x28e bytes) 000388 1788264980.677934 trunc from 0x0000dd to 0x0003a2 (0x2c6 bytes) 000389 1788264980.702363 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a3): Mirror 10: 0x88979862 (stale) Mirror 11: 0x88979862 (stale) Mirror 12: 0x88979862 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3): Mirror 10: 0x88979862 (stale) Mirror 11: 0x88979862 (stale) Mirror 12: 0x88979862 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3: skipped 000390 1788264981.070280 fallocate 0x00ca81 thru 0x0110f4 (0x4674 bytes) 000391 1788264981.076607 fallocate 0x001229 thru 0x00eb7c (0xd954 bytes) 000392 1788264981.080974 trunc from 0x0110f5 to 0x01060e (0xae6 bytes) 000393 1788264981.100120 trunc from 0x01060f to 0x02f1e2 (0x1ebd4 bytes) 000394 1788264981.127318 punch from 0x0263a5 to 0x02f1e2 (0x8e3e bytes) 000395 1788264981.132314 read 0x017d33 thru 0x01995e (0x1c2c bytes) 000397 1788264981.139269 fallocate 0x02caa2 thru 0x034b09 (0x8068 bytes) 000399 1788264981.151733 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34b0a): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b0a): Mirror 10: 0xf7180169 (stale) Mirror 11: 0xf7180169 (stale) Mirror 12: 0x292ec72d (stale) CRC-32 checksum value for chunk [0, 0x34b0a): Mirror 1: 0xdc74a179 Mirror 13: 0xdc74a179 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b0a: skipped 000400 1788264982.180994 punch from 0x0223b6 to 0x028394 (0x5fdf bytes) 000401 1788264982.241112 MIRROR_RESYNC 5 mirrors 000402 1788264983.382955 punch from 0x02e1b1 to 0x034b09 (0x6959 bytes) 000404 1788264983.438790 write 0x00e5f8 thru 0x011fc1 (0x39ca bytes) 000405 1788264983.463987 mapwrite 0x03e016 thru 0x03ffff (0x1fea bytes) 000406 1788264983.558369 fallocate 0x018ad6 thru 0x01cf2d (0x4458 bytes) 000407 1788264983.563855 trunc from 0x03023a to 0x034c2e (0x49f5 bytes) 000408 1788264983.568645 mapwrite 0x036c53 thru 0x03fe96 (0x9244 bytes) 000409 1788264983.630108 read 0x01458c thru 0x0238d0 (0xf345 bytes) 000410 1788264983.645577 write 0x0103f1 thru 0x018360 (0x7f70 bytes) 000411 1788264983.648301 read 0x0039bd thru 0x0051bf (0x1803 bytes) 000412 1788264983.663679 punch from 0x0092f7 to 0x009b1a (0x824 bytes) 000414 1788264983.674796 mapwrite 0x031561 thru 0x03ffff (0xea9f bytes) 000415 1788264983.714939 mapread 0x034e46 thru 0x036d2a (0x1ee5 bytes) 000416 1788264983.717346 punch from 0x00969e to 0x0190df (0xfa42 bytes) 000417 1788264983.733639 write 0x018eab thru 0x0260ec (0xd242 bytes) 000418 1788264983.742205 write 0x0339a7 thru 0x03ffff (0xc659 bytes) 000419 1788264983.743562 mapwrite 0x026cee thru 0x031817 (0xab2a bytes) 000420 1788264983.764467 trunc from 0x040000 to 0x01e41e (0x21be1 bytes) 000421 1788264983.798235 trunc from 0x01e41f to 0x00e2ed (0x10131 bytes) 000422 1788264983.857472 MIRROR_RESYNC 5 mirrors 000423 1788264984.731563 fallocate 0x02af76 thru 0x02ef74 (0x3fff bytes) 000424 1788264984.795521 read 0x00453d thru 0x00e2ed (0x9db1 bytes) 000426 1788264984.806097 fallocate 0x023b7d thru 0x02683a (0x2cbe bytes) 000427 1788264984.814273 read 0x0178ff thru 0x023508 (0xbc0a bytes) 000428 1788264984.832457 trunc from 0x017ca3 to 0x01f323 (0x7681 bytes) 000429 1788264984.840509 read 0x021196 thru 0x02683a (0x56a5 bytes) 000430 1788264984.841982 mapwrite 0x03a631 thru 0x03ffff (0x59cf bytes) 000432 1788264984.893542 write 0x02ef46 thru 0x03056f (0x162a bytes) 000433 1788264984.894565 write 0x03af56 thru 0x03ffff (0x50aa bytes) 000434 1788264984.895500 trunc from 0x01a9d1 to 0x02a792 (0xfdc2 bytes) 000435 1788264984.907325 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xe20eea22 (stale) Mirror 12: 0xe20eea22 (stale) Mirror 13: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000436 1788264985.380115 write 0x01ac52 thru 0x024142 (0x94f1 bytes) 000437 1788264985.403880 trunc from 0x025e2b to 0x026a24 (0xbfa bytes) 000438 1788264985.409804 trunc from 0x040000 to 0x030d0d (0xf2f2 bytes) 000439 1788264985.442881 punch from 0x007b86 to 0x00ba53 (0x3ece bytes) 000440 1788264985.450679 punch from 0x02c5ec to 0x030d0d (0x4722 bytes) 000441 1788264985.455255 mapwrite 0x01afea thru 0x01c062 (0x1079 bytes) 000442 1788264985.480072 MIRROR_RESYNC 4 mirrors 000443 1788264986.647335 fallocate 0x02c311 thru 0x02f013 (0x2d03 bytes) 000444 1788264986.689887 punch from 0x026b2f to 0x030d0d (0xa1df bytes) 000445 1788264986.694839 fallocate 0x0230c4 thru 0x02b1e7 (0x8124 bytes) 000446 1788264986.698966 read 0x009132 thru 0x0139d7 (0xa8a6 bytes) 000447 1788264986.709076 trunc from 0x006414 to 0x012b67 (0xc754 bytes) 000450 1788264986.714678 write 0x03be67 thru 0x03ffff (0x4199 bytes) 000451 1788264986.716080 read 0x0223a6 thru 0x0300fe (0xdd59 bytes) 000452 1788264986.726129 read 0x02f73b thru 0x03ca00 (0xd2c6 bytes) 000453 1788264986.727077 read 0x02beff thru 0x03a36f (0xe471 bytes) 000454 1788264986.727875 fallocate 0x007643 thru 0x014337 (0xccf5 bytes) 000455 1788264986.732132 write 0x000c3a thru 0x00e0b5 (0xd47c bytes) 000456 1788264986.734680 mapread 0x016dcc thru 0x017241 (0x476 bytes) 000459 1788264986.736500 punch from 0x023ef6 to 0x031de3 (0xdeee bytes) 000460 1788264986.741706 mapread 0x00b0c5 thru 0x00d71d (0x2659 bytes) 000461 1788264986.743326 trunc from 0x040000 to 0x03956e (0x6a91 bytes) 000462 1788264986.761495 fallocate 0x03843c thru 0x03ffff (0x7bc4 bytes) 000463 1788264986.768676 trunc from 0x020aaa to 0x028c9c (0x81f3 bytes) 000464 1788264986.774539 trunc from 0x040000 to 0x013534 (0x2cacb bytes) 000465 1788264986.795919 write 0x027488 thru 0x027b32 (0x6ab bytes) 000466 1788264986.797789 read 0x01b0fe thru 0x025a2a (0xa92d bytes) 000468 1788264986.807187 fallocate 0x03bf82 thru 0x03db77 (0x1bf6 bytes) 000469 1788264986.812672 punch from 0x0189a2 to 0x01d3ea (0x4a49 bytes) 000471 1788264986.819628 read 0x024d49 thru 0x027b32 (0x2dea bytes) 000472 1788264986.820665 mapread 0x017da7 thru 0x027b32 (0xfd8c bytes) 000473 1788264986.828586 mapread 0x00bedd thru 0x010f33 (0x5057 bytes) 000474 1788264986.830523 trunc from 0x00a611 to 0x013ce9 (0x96d9 bytes) 000475 1788264986.848644 mapwrite 0x00b74b thru 0x019da7 (0xe65d bytes) 000476 1788264986.941691 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27b33): Mirror 12: 0x9b1a7edc (stale) Mirror 13: 0x9b1a7edc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b33): Mirror 12: 0x9b1a7edc (stale) Mirror 13: 0x9b1a7edc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b33: skipped 000477 1788264987.522994 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27b33): Mirror 12: 0x9b1a7edc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b33): Mirror 12: 0x9b1a7edc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b33: skipped 000478 1788264987.996236 trunc from 0x00d1ff to 0x015ca2 (0x8aa4 bytes) 000479 1788264988.001765 fallocate 0x00ce2f thru 0x01392f (0x6b01 bytes) 000480 1788264988.010631 trunc from 0x027b33 to 0x03dabd (0x15f8b bytes) 000481 1788264988.031754 mapread 0x02dcf8 thru 0x030b3f (0x2e48 bytes) 000482 1788264988.054695 punch from 0x020b66 to 0x021f6a (0x1405 bytes) 000483 1788264988.061685 MIRROR_RESYNC 2 mirrors 000484 1788264988.632509 write 0x001025 thru 0x004754 (0x3730 bytes) 000485 1788264988.674161 write 0x027046 thru 0x027924 (0x8df bytes) 000486 1788264988.679500 write 0x015255 thru 0x018fbc (0x3d68 bytes) 000487 1788264988.688317 write 0x004ad8 thru 0x00eef9 (0xa422 bytes) 000488 1788264988.694927 fallocate 0x002aa2 thru 0x004472 (0x19d1 bytes) 000489 1788264988.699751 trunc from 0x03dabe to 0x02eb44 (0xef79 bytes) 000490 1788264988.743128 trunc from 0x02eb45 to 0x00a039 (0x24b0b bytes) 000491 1788264988.755256 trunc from 0x00a03a to 0x03487d (0x2a844 bytes) 000492 1788264988.771321 fallocate 0x034948 thru 0x03ffff (0xb6b8 bytes) 000493 1788264988.776399 write 0x03135a thru 0x035ec4 (0x4b6b bytes) 000494 1788264988.787154 read 0x0227c9 thru 0x03095b (0xe193 bytes) 000495 1788264988.798914 MIRROR_RESYNC 2 mirrors 000496 1788264989.476509 trunc from 0x025c42 to 0x030fda (0xb399 bytes) 000497 1788264989.518743 trunc from 0x008cc2 to 0x012ffd (0xa33c bytes) 000499 1788264989.530105 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) 000500 1788264989.676315 fallocate 0x02ed1e thru 0x03404f (0x5332 bytes) 000503 1788264989.684292 trunc from 0x024ca4 to 0x02c1e2 (0x753f bytes) 000504 1788264989.689826 read 0x018dd1 thru 0x020c0c (0x7e3c bytes) 000506 1788264989.698894 mapread 0x01a396 thru 0x01d4c5 (0x3130 bytes) 000510 1788264989.703442 fallocate 0x004f20 thru 0x007e3f (0x2f20 bytes) 000511 1788264989.708149 mapread 0x0391fe thru 0x03ffff (0x6e02 bytes) 000512 1788264989.725700 fallocate 0x0237a1 thru 0x025ec5 (0x2725 bytes) 000513 1788264989.732229 read 0x0048b4 thru 0x012474 (0xdbc1 bytes) 000514 1788264989.748130 fallocate 0x02125a thru 0x02da6d (0xc814 bytes) 000515 1788264989.751943 punch from 0x013df5 to 0x0195f9 (0x5805 bytes) 000516 1788264989.755888 fallocate 0x03ad34 thru 0x03ffff (0x52cc bytes) 000517 1788264989.759917 mapwrite 0x0212e6 thru 0x02dc12 (0xc92d bytes) 000518 1788264989.777070 mapread 0x0242f9 thru 0x03182b (0xd533 bytes) 000519 1788264989.777994 punch from 0x03cfd3 to 0x03ffff (0x302d bytes) 000520 1788264989.781929 punch from 0x00341f to 0x00d130 (0x9d12 bytes) 000522 1788264989.787202 write 0x0214e0 thru 0x0254c8 (0x3fe9 bytes) 000523 1788264989.792628 trunc from 0x02a61e to 0x0379e1 (0xd3c4 bytes) 000525 1788264989.798021 write 0x016e51 thru 0x02695f (0xfb0f bytes) 000526 1788264989.806764 MIRROR_EXTEND to 2 mirrors 000527 1788264990.542361 mapwrite 0x009949 thru 0x012ef5 (0x95ad bytes) 000529 1788264990.675263 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8104e7ee (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x37f4c72b Mirror 3: 0x37f4c72b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000530 1788264991.792740 trunc from 0x0150e6 to 0x016e52 (0x1d6d bytes) 000531 1788264991.861905 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8104e7ee (stale) Mirror 3: 0x37f4c72b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x91896a32 Mirror 4: 0x91896a32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000532 1788264993.142461 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8104e7ee (stale) Mirror 3: 0x37f4c72b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8104e7ee (stale) Mirror 3: 0x37f4c72b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000533 1788264993.532517 mapwrite 0x00c444 thru 0x014206 (0x7dc3 bytes) 000534 1788264993.589730 mapread 0x03a6a0 thru 0x03ffff (0x5960 bytes) 000535 1788264993.601158 fallocate 0x0309c7 thru 0x030ffa (0x634 bytes) 000536 1788264993.606219 mapread 0x02fd8d thru 0x034bcb (0x4e3f bytes) 000537 1788264993.621353 trunc from 0x012b04 to 0x02199e (0xee9b bytes) 000538 1788264993.630477 read 0x022d39 thru 0x024165 (0x142d bytes) 000540 1788264993.637698 write 0x0319b9 thru 0x03fa97 (0xe0df bytes) 000541 1788264993.645292 write 0x0102ea thru 0x01cdfc (0xcb13 bytes) 000542 1788264993.653204 read 0x015344 thru 0x01ec73 (0x9930 bytes) 000543 1788264993.664168 punch from 0x02fbed to 0x030582 (0x996 bytes) 000544 1788264993.668184 fallocate 0x009842 thru 0x013751 (0x9f10 bytes) 000545 1788264993.673646 punch from 0x003297 to 0x00aa1e (0x7788 bytes) 000546 1788264993.682046 MIRROR_RESYNC 3 mirrors 000547 1788264994.653383 trunc from 0x040000 to 0x03e6ed (0x1912 bytes) 000548 1788264994.700272 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e6ee): Mirror 3: 0xd29b4a71 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6ee): Mirror 3: 0xd29b4a71 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6ee: skipped 000549 1788264994.958540 trunc from 0x03e6ee to 0x00f3f1 (0x2f2fc bytes) 000551 1788264994.983644 fallocate 0x02bed4 thru 0x030f0c (0x5039 bytes) 000552 1788264994.988207 fallocate 0x00d207 thru 0x012fc8 (0x5dc2 bytes) 000553 1788264994.992841 mapwrite 0x038bfd thru 0x03ffff (0x7403 bytes) 000554 1788264995.043501 write 0x03e17d thru 0x03ffff (0x1e83 bytes) 000555 1788264995.044603 read 0x002388 thru 0x003ca9 (0x1922 bytes) 000556 1788264995.054457 read 0x01d510 thru 0x029ed5 (0xc9c6 bytes) 000557 1788264995.076186 fallocate 0x02c021 thru 0x03a132 (0xe112 bytes) 000558 1788264995.087545 punch from 0x0367d3 to 0x03ffff (0x982d bytes) 000559 1788264995.108218 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x742d020c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3175818c Mirror 4: 0x3175818c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000560 1788264996.083613 trunc from 0x003c61 to 0x012cdd (0xf07d bytes) 000562 1788264996.136686 write 0x0321c6 thru 0x0391c4 (0x6fff bytes) 000563 1788264996.153048 mapwrite 0x02b17b thru 0x039a9b (0xe921 bytes) 000564 1788264996.188172 trunc from 0x00c259 to 0x0146dd (0x8485 bytes) 000565 1788264996.193159 mapwrite 0x0006b5 thru 0x00b654 (0xafa0 bytes) 000566 1788264996.232431 trunc from 0x0341c9 to 0x03ffff (0xbe37 bytes) 000567 1788264996.237617 read 0x02f012 thru 0x030605 (0x15f4 bytes) 000569 1788264996.239763 write 0x02e77a thru 0x031baf (0x3436 bytes) 000570 1788264996.241410 trunc from 0x01b969 to 0x0281f8 (0xc890 bytes) 000571 1788264996.246901 read 0x03a281 thru 0x03ffff (0x5d7f bytes) 000572 1788264996.289552 punch from 0x02fc0d to 0x03f3dd (0xf7d1 bytes) 000573 1788264996.306239 mapread 0x02d7f3 thru 0x02f5b2 (0x1dc0 bytes) 000574 1788264996.317900 read 0x017916 thru 0x01c100 (0x47eb bytes) 000575 1788264996.321530 read 0x0213ba thru 0x02193e (0x585 bytes) 000576 1788264996.325450 mapwrite 0x0039da thru 0x0083ed (0x4a14 bytes) 000577 1788264996.366049 mapread 0x023f5c thru 0x02bb55 (0x7bfa bytes) 000578 1788264996.370412 mapwrite 0x03b308 thru 0x03ffff (0x4cf8 bytes) 000579 1788264996.445704 trunc from 0x018258 to 0x01f535 (0x72de bytes) 000580 1788264996.456899 trunc from 0x02c823 to 0x02deeb (0x16c9 bytes) 000582 1788264996.466899 trunc from 0x040000 to 0x00f71d (0x308e2 bytes) 000583 1788264996.502553 mapwrite 0x036243 thru 0x0364fa (0x2b8 bytes) 000584 1788264996.538188 mapread 0x020fa8 thru 0x0246b5 (0x370e bytes) 000585 1788264996.565559 punch from 0x02045f to 0x0221c0 (0x1d62 bytes) 000586 1788264996.576251 trunc from 0x02868d to 0x02c0bf (0x3a33 bytes) 000587 1788264996.586239 mapread 0x0307ab thru 0x0364fa (0x5d50 bytes) 000588 1788264996.618365 write 0x024826 thru 0x0258e1 (0x10bc bytes) 000589 1788264996.625189 trunc from 0x0364fb to 0x00d0e7 (0x29413 bytes) 000590 1788264996.640910 trunc from 0x00ae85 to 0x00bd96 (0xf12 bytes) 000591 1788264996.656364 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xd0e8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd0e8): Mirror 3: 0x163d19d6 (stale) Mirror 4: 0x163d19d6 (stale) CRC-32 checksum value for chunk [0, 0xd0e8): Mirror 1: 0xd1a5c5b1 Mirror 5: 0xd1a5c5b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd0e8: skipped 000592 1788264997.555683 mapwrite 0x02bed9 thru 0x035508 (0x9630 bytes) 000593 1788264997.757068 write 0x0142e8 thru 0x0167ac (0x24c5 bytes) 000594 1788264997.789002 trunc from 0x03333f to 0x035508 (0x21ca bytes) 000595 1788264997.800667 mapwrite 0x009bdd thru 0x014dff (0xb223 bytes) 000596 1788264997.865045 trunc from 0x035509 to 0x009e6b (0x2b69d bytes) 000597 1788264997.916148 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9e6c): Mirror 3: 0x9cdb4f63 (stale) Mirror 4: 0x9cdb4f63 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9e6c): Mirror 3: 0x9cdb4f63 (stale) Mirror 4: 0x9cdb4f63 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9e6c: skipped 000598 1788264998.453864 mapread 0x001f1f thru 0x009e6b (0x7f4d bytes) 000600 1788264998.478017 read 0x008d06 thru 0x009e6b (0x1166 bytes) 000601 1788264998.479427 fallocate 0x017569 thru 0x01fb1c (0x85b4 bytes) 000602 1788264998.487341 trunc from 0x009e6c to 0x022f13 (0x190a8 bytes) 000603 1788264998.514479 mapwrite 0x03d1b7 thru 0x03ffff (0x2e49 bytes) 000604 1788264998.580171 write 0x0007e8 thru 0x0041ca (0x39e3 bytes) 000605 1788264998.599520 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x742d020c (stale) Mirror 4: 0x3175818c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x621d3010 Mirror 5: 0x621d3010 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000607 1788264999.743528 punch from 0x03429b to 0x03ffff (0xbd65 bytes) 000608 1788264999.831544 fallocate 0x02b7c0 thru 0x0325d7 (0x6e18 bytes) 000609 1788264999.849647 mapread 0x02e294 thru 0x02e944 (0x6b1 bytes) 000610 1788264999.864490 fallocate 0x0361a6 thru 0x03ffff (0x9e5a bytes) 000611 1788264999.893202 trunc from 0x01054d to 0x016d71 (0x6825 bytes) 000612 1788264999.898464 trunc from 0x02ec86 to 0x031741 (0x2abc bytes) 000613 1788264999.920187 mapread 0x006d1b thru 0x00e3b6 (0x769c bytes) 000615 1788264999.951047 punch from 0x027ab8 to 0x036585 (0xeace bytes) 000616 1788264999.956462 punch from 0x018d45 to 0x01ce04 (0x40c0 bytes) 000617 1788264999.960512 trunc from 0x040000 to 0x027c9c (0x18363 bytes) 000618 1788265000.008856 read 0x01cbc1 thru 0x01e4a8 (0x18e8 bytes) 000619 1788265000.026762 mapread 0x015728 thru 0x016bc7 (0x14a0 bytes) 000620 1788265000.037880 punch from 0x020891 to 0x025e50 (0x55c0 bytes) 000621 1788265000.042132 mapwrite 0x02552b thru 0x0284a5 (0x2f7b bytes) 000622 1788265000.093503 read 0x0081eb thru 0x01268d (0xa4a3 bytes) 000623 1788265000.095589 read 0x002e73 thru 0x00cdd6 (0x9f64 bytes) 000626 1788265000.109795 read 0x01a0a2 thru 0x02044b (0x63aa bytes) 000627 1788265000.111211 trunc from 0x0284a6 to 0x00c082 (0x1c423 bytes) 000628 1788265000.129499 trunc from 0x00c083 to 0x0057a5 (0x68dd bytes) 000629 1788265000.146512 fallocate 0x02b924 thru 0x0385c7 (0xcca4 bytes) 000630 1788265000.153085 trunc from 0x0057a6 to 0x007948 (0x21a3 bytes) 000632 1788265000.187249 fallocate 0x00009f thru 0x00414d (0x40af bytes) 000633 1788265000.209679 mapwrite 0x0039a5 thru 0x00dd7a (0xa3d6 bytes) 000634 1788265000.260984 write 0x025c59 thru 0x029309 (0x36b1 bytes) 000635 1788265000.264629 punch from 0x01f6e5 to 0x025900 (0x621c bytes) 000636 1788265000.278653 mapread 0x013b43 thru 0x01bbc6 (0x8084 bytes) 000637 1788265000.298565 read 0x003a14 thru 0x003d99 (0x386 bytes) 000638 1788265000.299231 fallocate 0x03a115 thru 0x03ffff (0x5eeb bytes) 000639 1788265000.311979 punch from 0x000b08 to 0x004c6b (0x4164 bytes) 000640 1788265000.323065 MIRROR_RESYNC 4 mirrors 000641 1788265001.663538 trunc from 0x02930a to 0x026848 (0x2ac1 bytes) 000642 1788265001.730191 MIRROR_RESYNC 4 mirrors 000643 1788265002.917007 mapread 0x009fb6 thru 0x00b217 (0x1262 bytes) 000644 1788265002.938563 mapread 0x00b8a3 thru 0x0196c8 (0xde26 bytes) 000645 1788265002.956998 write 0x0033c0 thru 0x01286f (0xf4b0 bytes) 000646 1788265003.002673 mapread 0x012dae thru 0x0175a1 (0x47f4 bytes) 000648 1788265003.029150 mapread 0x00154e thru 0x009feb (0x8a9e bytes) 000649 1788265003.055464 mapread 0x002d4e thru 0x0084c1 (0x5774 bytes) 000651 1788265003.066657 write 0x028e64 thru 0x030a78 (0x7c15 bytes) 000652 1788265003.069134 read 0x025020 thru 0x02b5e5 (0x65c6 bytes) 000653 1788265003.086529 trunc from 0x030a79 to 0x03e354 (0xd8dc bytes) 000654 1788265003.135754 trunc from 0x002341 to 0x004a8a (0x274a bytes) 000655 1788265003.166674 read 0x02223a thru 0x0258d2 (0x3699 bytes) 000656 1788265003.184200 mapread 0x013bf5 thru 0x021f9e (0xe3aa bytes) 000657 1788265003.185730 fallocate 0x0232ff thru 0x02d9da (0xa6dc bytes) 000658 1788265003.199214 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e355): Mirror 4: 0xe0d93222 (stale) Mirror 5: 0xe0d93222 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e355): Mirror 4: 0xe0d93222 (stale) Mirror 5: 0xe0d93222 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e355: skipped 000660 1788265003.484208 punch from 0x009f03 to 0x0171a3 (0xd2a1 bytes) 000661 1788265003.489753 mapwrite 0x03c03a thru 0x03ffff (0x3fc6 bytes) 000662 1788265003.587168 punch from 0x037158 to 0x03f7f7 (0x86a0 bytes) 000663 1788265003.597525 punch from 0x02ca72 to 0x0337c9 (0x6d58 bytes) 000665 1788265003.608375 mapread 0x001f8e thru 0x009e63 (0x7ed6 bytes) 000666 1788265003.628737 mapread 0x009d8e thru 0x00dee6 (0x4159 bytes) 000667 1788265003.637079 MIRROR_RESYNC 3 mirrors 000669 1788265004.836752 trunc from 0x040000 to 0x02dc68 (0x12397 bytes) 000670 1788265004.916579 MIRROR_RESYNC 3 mirrors 000671 1788265006.058545 write 0x005e97 thru 0x00e6c3 (0x882d bytes) 000672 1788265006.126384 trunc from 0x0185ed to 0x01be99 (0x38ad bytes) 000675 1788265006.132019 mapread 0x0235c6 thru 0x02dc68 (0xa6a3 bytes) 000676 1788265006.156103 mapread 0x020b45 thru 0x02dc68 (0xd124 bytes) 000677 1788265006.158664 mapwrite 0x03ef8d thru 0x03ffff (0x1073 bytes) 000678 1788265006.218450 MIRROR_RESYNC 3 mirrors 000679 1788265007.320478 write 0x01c9b7 thru 0x020e35 (0x447f bytes) 000680 1788265007.368472 punch from 0x01972b to 0x01cd7e (0x3654 bytes) 000681 1788265007.384626 fallocate 0x0028d3 thru 0x008527 (0x5c55 bytes) 000682 1788265007.390538 punch from 0x0284fa to 0x02e121 (0x5c28 bytes) 000683 1788265007.395043 punch from 0x011b70 to 0x012f10 (0x13a1 bytes) 000684 1788265007.399010 trunc from 0x040000 to 0x0350fe (0xaf01 bytes) 000686 1788265007.438308 mapwrite 0x010f6a thru 0x019401 (0x8498 bytes) 000687 1788265007.495293 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x350ff): Mirror 5: 0xb1eea13d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x350ff): Mirror 5: 0xb1eea13d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x350ff: skipped 000688 1788265007.815843 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) 000689 1788265007.974238 mapread 0x00b50a thru 0x019ff2 (0xeae9 bytes) 000690 1788265007.998207 write 0x00ac02 thru 0x015d5b (0xb15a bytes) 000692 1788265008.005671 trunc from 0x0350ff to 0x03db5a (0x8a5c bytes) 000693 1788265008.019481 trunc from 0x03db5b to 0x0041fa (0x39960 bytes) 000694 1788265008.037312 trunc from 0x0041fb to 0x005dcb (0x1bd1 bytes) 000695 1788265008.052094 trunc from 0x005b7f to 0x005dcb (0x24d bytes) 000696 1788265008.056708 read 0x003ad4 thru 0x005dcb (0x22f8 bytes) 000697 1788265008.061687 write 0x02351b thru 0x029306 (0x5dec bytes) 000698 1788265008.063309 read 0x014125 thru 0x023894 (0xf770 bytes) 000699 1788265008.070616 punch from 0x015234 to 0x019361 (0x412e bytes) 000702 1788265008.076295 trunc from 0x01b8b6 to 0x029306 (0xda51 bytes) 000703 1788265008.089453 mapwrite 0x00fcdf thru 0x019895 (0x9bb7 bytes) 000704 1788265008.128138 trunc from 0x029307 to 0x0295bb (0x2b5 bytes) 000705 1788265008.139854 punch from 0x00f66c to 0x019422 (0x9db7 bytes) 000706 1788265008.143940 punch from 0x002c29 to 0x006a85 (0x3e5d bytes) 000708 1788265008.153162 write 0x0211aa thru 0x026983 (0x57da bytes) 000709 1788265008.168427 MIRROR_EXTEND to 2 mirrors 000710 1788265008.994823 read 0x0021fd thru 0x011900 (0xf704 bytes) 000711 1788265009.035149 trunc from 0x0295bc to 0x013bb1 (0x15a0a bytes) 000712 1788265009.135065 mapread 0x006dc6 thru 0x01255e (0xb799 bytes) 000713 1788265009.160693 fallocate 0x02e939 thru 0x03abac (0xc274 bytes) 000714 1788265009.166560 punch from 0x009f3f to 0x012204 (0x82c6 bytes) 000715 1788265009.171701 mapwrite 0x005afb thru 0x008c2b (0x3131 bytes) 000716 1788265009.195208 trunc from 0x00ddce to 0x013bb1 (0x5de4 bytes) 000717 1788265009.201765 punch from 0x0043db to 0x013bb1 (0xf7d7 bytes) 000719 1788265009.212978 trunc from 0x013bb2 to 0x02fd7d (0x1c1cc bytes) 000720 1788265009.235813 write 0x0098bb thru 0x009d91 (0x4d7 bytes) 000721 1788265009.247581 mapwrite 0x001bdf thru 0x0033b7 (0x17d9 bytes) 000722 1788265009.267706 fallocate 0x005f8d thru 0x0066ac (0x720 bytes) 000723 1788265009.272347 mapwrite 0x02e9c4 thru 0x0335d3 (0x4c10 bytes) 000725 1788265009.320737 mapwrite 0x00e85d thru 0x01d398 (0xeb3c bytes) 000726 1788265009.383910 fallocate 0x006c39 thru 0x0097c9 (0x2b91 bytes) 000727 1788265009.394167 read 0x030e03 thru 0x0335d3 (0x27d1 bytes) 000728 1788265009.395687 trunc from 0x0335d4 to 0x031c53 (0x1980 bytes) 000729 1788265009.425196 fallocate 0x025503 thru 0x02ce19 (0x7917 bytes) 000730 1788265009.434382 trunc from 0x031c54 to 0x00c512 (0x25741 bytes) 000731 1788265009.477676 trunc from 0x006fea to 0x00c512 (0x5529 bytes) 000732 1788265009.488181 fallocate 0x010f53 thru 0x011a06 (0xab4 bytes) 000733 1788265009.495703 fallocate 0x03ae52 thru 0x03ffff (0x51ae bytes) 000734 1788265009.504124 write 0x012d51 thru 0x013424 (0x6d4 bytes) 000735 1788265009.507224 fallocate 0x00f82e thru 0x01b6ca (0xbe9d bytes) 000736 1788265009.515134 mapread 0x006304 thru 0x013095 (0xcd92 bytes) 000738 1788265009.536070 mapread 0x00b058 thru 0x016d1c (0xbcc5 bytes) 000740 1788265009.537746 mapwrite 0x034869 thru 0x03bd5e (0x74f6 bytes) 000742 1788265009.589263 punch from 0x022648 to 0x026d4e (0x4707 bytes) 000743 1788265009.594319 trunc from 0x03bd5f to 0x029544 (0x1281a bytes) 000744 1788265009.610055 punch from 0x026500 to 0x029544 (0x3045 bytes) 000745 1788265009.617461 fallocate 0x030aa8 thru 0x0394ec (0x8a45 bytes) 000746 1788265009.624963 trunc from 0x029545 to 0x00ad45 (0x1e7ff bytes) 000747 1788265009.652738 fallocate 0x0304e6 thru 0x030e34 (0x94f bytes) 000748 1788265009.657710 read 0x024cec thru 0x02699f (0x1cb4 bytes) 000749 1788265009.674414 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30e35): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e35): Mirror 2: 0x5e3b541a (stale) CRC-32 checksum value for chunk [0, 0x30e35): Mirror 1: 0x6d39c41c Mirror 3: 0x6d39c41c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e35: skipped 000750 1788265010.360413 mapread 0x00371d thru 0x007491 (0x3d75 bytes) 000751 1788265010.396724 write 0x0106c7 thru 0x012dbd (0x26f7 bytes) 000752 1788265010.472466 write 0x030048 thru 0x030241 (0x1fa bytes) 000753 1788265010.478264 trunc from 0x000901 to 0x00f2c9 (0xe9c9 bytes) 000754 1788265010.486590 fallocate 0x025e29 thru 0x02685b (0xa33 bytes) 000756 1788265010.491279 fallocate 0x02c34f thru 0x036102 (0x9db4 bytes) 000757 1788265010.495494 fallocate 0x00c93b thru 0x00d8dc (0xfa2 bytes) 000759 1788265010.501385 read 0x0129d6 thru 0x019446 (0x6a71 bytes) 000760 1788265010.515164 mapread 0x01b9f2 thru 0x01fc8a (0x4299 bytes) 000761 1788265010.538518 trunc from 0x01d65b to 0x02d551 (0xfef7 bytes) 000764 1788265010.566033 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36103): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36103): Mirror 2: 0x9283ddd0 (stale) Mirror 3: 0xbad0cadf (stale) CRC-32 checksum value for chunk [0, 0x36103): Mirror 1: 0xbda69793 Mirror 4: 0xbda69793 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36103: skipped 000765 1788265011.700915 trunc from 0x036103 to 0x00dad3 (0x2862f bytes) 000766 1788265011.774919 trunc from 0x00dad4 to 0x03f14f (0x3167c bytes) 000767 1788265011.815691 read 0x02d69b thru 0x035835 (0x819b bytes) 000768 1788265011.830999 mapwrite 0x023830 thru 0x02fb6f (0xc340 bytes) 000769 1788265011.892060 mapread 0x03d598 thru 0x03f14f (0x1bb8 bytes) 000770 1788265011.909969 trunc from 0x03f150 to 0x035894 (0x98bb bytes) 000771 1788265011.943377 read 0x01154a thru 0x012a80 (0x1537 bytes) 000772 1788265011.965334 fallocate 0x00c085 thru 0x01bfb8 (0xff34 bytes) 000773 1788265011.977738 trunc from 0x035895 to 0x001210 (0x34684 bytes) 000774 1788265012.014848 mapread 0x000d5f thru 0x001210 (0x4b2 bytes) 000776 1788265012.019011 trunc from 0x001211 to 0x00b039 (0x9e29 bytes) 000777 1788265012.059936 MIRROR_RESYNC 4 mirrors 000778 1788265012.860974 mapwrite 0x01e059 thru 0x02db79 (0xfb21 bytes) 000779 1788265012.975628 read 0x018049 thru 0x021592 (0x954a bytes) 000780 1788265012.988596 mapwrite 0x023391 thru 0x025691 (0x2301 bytes) 000782 1788265013.020608 MIRROR_RESYNC 4 mirrors 000783 1788265013.930392 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) 000784 1788265014.257322 mapwrite 0x01d491 thru 0x020a0c (0x357c bytes) 000785 1788265014.340724 read 0x029bde thru 0x02b065 (0x1488 bytes) 000786 1788265014.347499 trunc from 0x00702d to 0x00ca02 (0x59d6 bytes) 000787 1788265014.352009 write 0x01595b thru 0x01de7f (0x8525 bytes) 000788 1788265014.362559 trunc from 0x00e3c9 to 0x01e03f (0xfc77 bytes) 000789 1788265014.380429 MIRROR_RESYNC 3 mirrors 000790 1788265014.937759 fallocate 0x02f91b thru 0x036265 (0x694b bytes) 000791 1788265014.986969 mapwrite 0x02ac14 thru 0x031846 (0x6c33 bytes) 000793 1788265015.119991 fallocate 0x00299e thru 0x00562c (0x2c8f bytes) 000794 1788265015.134097 punch from 0x03076a to 0x031846 (0x10dd bytes) 000795 1788265015.140821 punch from 0x01c074 to 0x026018 (0x9fa5 bytes) 000796 1788265015.145414 fallocate 0x01afdc thru 0x01c486 (0x14ab bytes) 000797 1788265015.153405 punch from 0x003eda to 0x00d8a6 (0x99cd bytes) 000798 1788265015.173733 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31847): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31847): Mirror 3: 0x5922a1fc (stale) Mirror 4: 0x5922a1fc (stale) CRC-32 checksum value for chunk [0, 0x31847): Mirror 1: 0xa55eab61 Mirror 5: 0xa55eab61 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31847: skipped 000799 1788265015.998270 trunc from 0x015a2c to 0x022d15 (0xd2ea bytes) 000800 1788265016.065500 trunc from 0x016e6d to 0x0260d2 (0xf266 bytes) 000802 1788265016.075309 write 0x0246c7 thru 0x02debf (0x97f9 bytes) 000803 1788265016.101494 trunc from 0x031847 to 0x0183f4 (0x19452 bytes) 000805 1788265016.162741 mapread 0x000001 thru 0x005786 (0x5786 bytes) 000806 1788265016.184222 write 0x00e332 thru 0x013904 (0x55d3 bytes) 000807 1788265016.188756 write 0x001838 thru 0x00f565 (0xdd2e bytes) 000808 1788265016.194048 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x183f5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x183f5): Mirror 3: 0x1252b0ab (stale) Mirror 4: 0x1252b0ab (stale) Mirror 5: 0x1252b0ab (stale) CRC-32 checksum value for chunk [0, 0x183f5): Mirror 1: 0x7a52f5f Mirror 6: 0x7a52f5f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x183f5: skipped 000809 1788265016.995145 write 0x00a476 thru 0x017bab (0xd736 bytes) 000810 1788265017.042949 trunc from 0x0183f5 to 0x02d37c (0x14f88 bytes) 000811 1788265017.089612 trunc from 0x006a39 to 0x0136fd (0xccc5 bytes) 000812 1788265017.107541 trunc from 0x02d37d to 0x03f129 (0x11dad bytes) 000813 1788265017.121898 mapwrite 0x008415 thru 0x008e55 (0xa41 bytes) 000814 1788265017.138975 mapwrite 0x00a8ba thru 0x00c2f5 (0x1a3c bytes) 000815 1788265017.169577 read 0x0202d5 thru 0x021f8c (0x1cb8 bytes) 000816 1788265017.175570 read 0x01d387 thru 0x02a688 (0xd302 bytes) 000817 1788265017.193058 trunc from 0x03f12a to 0x02168e (0x1da9b bytes) 000818 1788265017.222447 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2168f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2168f): Mirror 3: 0x53a8a60d (stale) Mirror 4: 0x53a8a60d (stale) Mirror 5: 0x1cd460b4 (stale) Mirror 6: 0x8a5944b3 (stale) CRC-32 checksum value for chunk [0, 0x2168f): Mirror 1: 0xce679864 Mirror 7: 0xce679864 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2168f: skipped 000819 1788265017.995672 read 0x00e478 thru 0x016fd7 (0x8b60 bytes) 000820 1788265018.012230 trunc from 0x02168f to 0x01b950 (0x5d3e bytes) 000821 1788265018.108816 mapwrite 0x023884 thru 0x0284fa (0x4c77 bytes) 000822 1788265018.199464 mapread 0x012b7d thru 0x0227b8 (0xfc3c bytes) 000824 1788265018.235867 write 0x0301b1 thru 0x03347a (0x32ca bytes) 000825 1788265018.236987 trunc from 0x03347b to 0x00cae0 (0x2699a bytes) 000826 1788265018.277447 punch from 0x00bcf2 to 0x00cae0 (0xdef bytes) 000827 1788265018.285036 mapread 0x008a0d thru 0x00cae0 (0x40d4 bytes) 000828 1788265018.310892 mapwrite 0x002c37 thru 0x00a2ec (0x76b6 bytes) 000831 1788265018.339688 trunc from 0x00ad40 to 0x00cae0 (0x1da1 bytes) 000832 1788265018.345236 mapread 0x001c50 thru 0x00590e (0x3cbf bytes) 000833 1788265018.346594 trunc from 0x000f0d to 0x00850c (0x7600 bytes) 000834 1788265018.352044 fallocate 0x007c2c thru 0x016b5b (0xef30 bytes) 000835 1788265018.357960 read 0x000162 thru 0x00c319 (0xc1b8 bytes) 000836 1788265018.369592 fallocate 0x032003 thru 0x034f53 (0x2f51 bytes) 000837 1788265018.373182 trunc from 0x00cae1 to 0x025994 (0x18eb4 bytes) 000838 1788265018.386121 trunc from 0x0155cd to 0x01daa1 (0x84d5 bytes) 000839 1788265018.391435 punch from 0x023de0 to 0x025994 (0x1bb5 bytes) 000840 1788265018.397114 read 0x00016a thru 0x00f4d7 (0xf36e bytes) 000842 1788265018.406942 punch from 0x00ac4e to 0x018933 (0xdce6 bytes) 000843 1788265018.411323 trunc from 0x01477f to 0x01f3ff (0xac81 bytes) 000844 1788265018.416377 trunc from 0x000154 to 0x00aa91 (0xa93e bytes) 000845 1788265018.424684 MIRROR_RESYNC 6 mirrors 000847 1788265019.938079 mapwrite 0x03eebf thru 0x03ffff (0x1141 bytes) 000848 1788265020.012531 fallocate 0x031d56 thru 0x0380a1 (0x634c bytes) 000849 1788265020.039503 fallocate 0x03cb60 thru 0x03ffff (0x34a0 bytes) 000852 1788265020.061120 punch from 0x02ead0 to 0x03a660 (0xbb91 bytes) 000853 1788265020.095351 MIRROR_RESYNC 6 mirrors 000854 1788265021.277586 punch from 0x0180b3 to 0x021161 (0x90af bytes) 000855 1788265021.313460 punch from 0x028824 to 0x02e183 (0x5960 bytes) 000856 1788265021.317993 read 0x015869 thru 0x01ccee (0x7486 bytes) 000858 1788265021.325762 write 0x0251c8 thru 0x02ef32 (0x9d6b bytes) 000859 1788265021.340272 mapwrite 0x0188a2 thru 0x025853 (0xcfb2 bytes) 000860 1788265021.396367 MIRROR_RESYNC 6 mirrors 000861 1788265022.947334 punch from 0x007262 to 0x00dcaf (0x6a4e bytes) 000862 1788265022.980208 trunc from 0x040000 to 0x0236f6 (0x1c909 bytes) 000863 1788265022.997440 trunc from 0x0236f7 to 0x02d428 (0x9d32 bytes) 000864 1788265023.013375 mapwrite 0x00e038 thru 0x015403 (0x73cc bytes) 000866 1788265023.048420 fallocate 0x00e11a thru 0x00eb02 (0x9e9 bytes) 000867 1788265023.052804 trunc from 0x004e85 to 0x00e4d1 (0x964d bytes) 000868 1788265023.061149 read 0x02c84e thru 0x02d428 (0xbdb bytes) 000869 1788265023.066509 write 0x015c75 thru 0x01dcc7 (0x8053 bytes) 000870 1788265023.072082 fallocate 0x031f91 thru 0x034862 (0x28d2 bytes) 000871 1788265023.080547 trunc from 0x034863 to 0x008a8f (0x2bdd3 bytes) 000872 1788265023.110191 write 0x02ce66 thru 0x039e17 (0xcfb2 bytes) 000873 1788265023.111896 mapwrite 0x01670e thru 0x01933f (0x2c32 bytes) 000874 1788265023.142232 mapread 0x028afc thru 0x02a9af (0x1eb4 bytes) 000875 1788265023.151016 read 0x02d9d4 thru 0x02e178 (0x7a5 bytes) 000876 1788265023.151557 read 0x01d3cb thru 0x0233ff (0x6035 bytes) 000877 1788265023.160452 trunc from 0x035c4e to 0x039e17 (0x41ca bytes) 000878 1788265023.165449 punch from 0x01687f to 0x017218 (0x99a bytes) 000879 1788265023.170394 mapwrite 0x01f32f thru 0x02d7da (0xe4ac bytes) 000880 1788265023.190640 punch from 0x02b5f8 to 0x030c54 (0x565d bytes) 000881 1788265023.195612 fallocate 0x00a62f thru 0x0148e7 (0xa2b9 bytes) 000882 1788265023.204270 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x39e18): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e18): Mirror 3: 0x60ea8a22 (stale) Mirror 4: 0x60ea8a22 (stale) Mirror 5: 0x60ea8a22 (stale) Mirror 6: 0x60ea8a22 (stale) Mirror 7: 0x60ea8a22 (stale) CRC-32 checksum value for chunk [0, 0x39e18): Mirror 1: 0x86b5891e Mirror 8: 0x86b5891e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e18: skipped 000883 1788265024.332053 mapread 0x02ca81 thru 0x0349b2 (0x7f32 bytes) 000884 1788265024.358491 write 0x015d5f thru 0x01930a (0x35ac bytes) 000885 1788265024.396624 read 0x0231e6 thru 0x023e8b (0xca6 bytes) 000886 1788265024.400765 fallocate 0x011af3 thru 0x018715 (0x6c23 bytes) 000887 1788265024.404972 mapwrite 0x011047 thru 0x01cb44 (0xbafe bytes) 000888 1788265024.460573 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39e18): Mirror 3: 0x60ea8a22 (stale) Mirror 5: 0x60ea8a22 (stale) Mirror 6: 0x60ea8a22 (stale) Mirror 7: 0x60ea8a22 (stale) Mirror 8: 0x86b5891e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e18): Mirror 3: 0x60ea8a22 (stale) Mirror 5: 0x60ea8a22 (stale) Mirror 6: 0x60ea8a22 (stale) Mirror 7: 0x60ea8a22 (stale) Mirror 8: 0x86b5891e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e18: skipped 000889 1788265024.925483 mapwrite 0x03a2d7 thru 0x03ffff (0x5d29 bytes) 000890 1788265025.002664 write 0x029831 thru 0x037e18 (0xe5e8 bytes) 000891 1788265025.011759 fallocate 0x030c87 thru 0x03df44 (0xd2be bytes) 000892 1788265025.017567 punch from 0x009e92 to 0x00d4de (0x364d bytes) 000894 1788265025.023200 mapwrite 0x004623 thru 0x00747a (0x2e58 bytes) 000895 1788265025.048561 trunc from 0x017b58 to 0x025c95 (0xe13e bytes) 000896 1788265025.055031 read 0x015262 thru 0x01c5d2 (0x7371 bytes) 000897 1788265025.069179 read 0x02160e thru 0x02683d (0x5230 bytes) 000898 1788265025.080651 write 0x00c189 thru 0x011e9d (0x5d15 bytes) 000899 1788265025.082966 trunc from 0x040000 to 0x0340d2 (0xbf2d bytes) 000900 1788265025.098185 punch from 0x02d052 to 0x03313e (0x60ed bytes) 000901 1788265025.102961 punch from 0x022f56 to 0x029966 (0x6a11 bytes) 000902 1788265025.106865 trunc from 0x018e41 to 0x0205fe (0x77be bytes) 000903 1788265025.110588 read 0x005629 thru 0x0105ed (0xafc5 bytes) 000904 1788265025.111292 punch from 0x00d941 to 0x00eed0 (0x1590 bytes) 000905 1788265025.131830 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x340d3): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340d3): Mirror 3: 0xa038bae9 (stale) Mirror 5: 0xa038bae9 (stale) Mirror 6: 0xa038bae9 (stale) Mirror 7: 0xa038bae9 (stale) Mirror 8: 0x66f34c3b (stale) CRC-32 checksum value for chunk [0, 0x340d3): Mirror 1: 0x2d0e451a Mirror 9: 0x2d0e451a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340d3: skipped 000907 1788265026.205530 fallocate 0x035e55 thru 0x037d56 (0x1f02 bytes) 000908 1788265026.274869 trunc from 0x0166a7 to 0x02603c (0xf996 bytes) 000909 1788265026.281390 write 0x015ce1 thru 0x01e5b5 (0x88d5 bytes) 000910 1788265026.315059 MIRROR_RESYNC 7 mirrors 000911 1788265028.708695 punch from 0x01bb63 to 0x025f98 (0xa436 bytes) 000912 1788265028.753953 trunc from 0x003d48 to 0x010dbf (0xd078 bytes) 000913 1788265028.758716 trunc from 0x008d46 to 0x01147c (0x8737 bytes) 000914 1788265028.762740 read 0x02c724 thru 0x037c9e (0xb57b bytes) 000915 1788265028.778006 write 0x016063 thru 0x0192a4 (0x3242 bytes) 000916 1788265028.788041 trunc from 0x037d57 to 0x03cb23 (0x4dcd bytes) 000918 1788265028.805217 trunc from 0x002bc5 to 0x0056ce (0x2b0a bytes) 000919 1788265028.812804 read 0x03cacb thru 0x03cb23 (0x59 bytes) 000920 1788265028.839526 write 0x038546 thru 0x038fe9 (0xaa4 bytes) 000921 1788265028.843983 mapwrite 0x03cad5 thru 0x03ffff (0x352b bytes) 000922 1788265028.882156 fallocate 0x0349e5 thru 0x039402 (0x4a1e bytes) 000923 1788265028.887356 write 0x01bd44 thru 0x020962 (0x4c1f bytes) 000925 1788265028.889639 trunc from 0x040000 to 0x0137ff (0x2c800 bytes) 000926 1788265028.908940 fallocate 0x00a6b8 thru 0x00c4bb (0x1e04 bytes) 000927 1788265028.914196 read 0x002fac thru 0x00a2ec (0x7341 bytes) 000928 1788265028.915347 read 0x004f19 thru 0x0135ad (0xe695 bytes) 000929 1788265028.916408 punch from 0x0001cd to 0x008c63 (0x8a97 bytes) 000930 1788265028.920333 read 0x009714 thru 0x00acff (0x15ec bytes) 000931 1788265028.921400 trunc from 0x0133dd to 0x0137ff (0x423 bytes) 000932 1788265028.932308 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x13800): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13800): Mirror 3: 0xd8395376 (stale) Mirror 5: 0xd8395376 (stale) Mirror 6: 0xd8395376 (stale) Mirror 7: 0xd8395376 (stale) Mirror 8: 0xd8395376 (stale) Mirror 9: 0xd8395376 (stale) CRC-32 checksum value for chunk [0, 0x13800): Mirror 1: 0x55535e99 Mirror 10: 0x55535e99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13800: skipped 000933 1788265029.669509 trunc from 0x013800 to 0x008de7 (0xaa18 bytes) 000934 1788265029.737130 mapwrite 0x011880 thru 0x01991d (0x809e bytes) 000935 1788265029.848439 trunc from 0x01991e to 0x0014d7 (0x18446 bytes) 000937 1788265029.884246 mapread 0x000237 thru 0x0014d7 (0x12a1 bytes) 000938 1788265029.891512 write 0x001529 thru 0x00e75d (0xd235 bytes) 000939 1788265029.893932 mapwrite 0x029b45 thru 0x02fb81 (0x603d bytes) 000940 1788265029.960895 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fb82): Mirror 3: 0x108ac63 (stale) Mirror 6: 0x108ac63 (stale) Mirror 7: 0x108ac63 (stale) Mirror 8: 0x108ac63 (stale) Mirror 9: 0x108ac63 (stale) Mirror 10: 0xaba00567 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb82): Mirror 3: 0x108ac63 (stale) Mirror 6: 0x108ac63 (stale) Mirror 7: 0x108ac63 (stale) Mirror 8: 0x108ac63 (stale) Mirror 9: 0x108ac63 (stale) Mirror 10: 0xaba00567 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb82: skipped 000942 1788265030.606617 trunc from 0x00e71b to 0x00e960 (0x246 bytes) 000944 1788265030.617331 punch from 0x01dafb to 0x01f3f0 (0x18f6 bytes) 000945 1788265030.634008 trunc from 0x02ee74 to 0x02fb81 (0xd0e bytes) 000946 1788265030.640721 punch from 0x024b6e to 0x02fa15 (0xaea8 bytes) 000947 1788265030.654038 write 0x017902 thru 0x01c827 (0x4f26 bytes) 000950 1788265030.680615 trunc from 0x02fb82 to 0x01c7de (0x133a3 bytes) 000951 1788265030.706592 fallocate 0x01e4d3 thru 0x025ab8 (0x75e6 bytes) 000952 1788265030.722679 trunc from 0x00122b to 0x0079b3 (0x6789 bytes) 000954 1788265030.728399 mapread 0x012472 thru 0x017166 (0x4cf5 bytes) 000955 1788265030.749535 fallocate 0x00f3b9 thru 0x01e5e1 (0xf229 bytes) 000956 1788265030.756485 trunc from 0x01c7df to 0x0335b5 (0x16dd7 bytes) 000957 1788265030.772513 fallocate 0x005164 thru 0x006c63 (0x1b00 bytes) 000958 1788265030.777278 write 0x011a86 thru 0x015d3a (0x42b5 bytes) 000959 1788265030.779140 write 0x035849 thru 0x03ffff (0xa7b7 bytes) 000960 1788265030.780980 punch from 0x02aac1 to 0x0377c2 (0xcd02 bytes) 000961 1788265030.800522 MIRROR_RESYNC 7 mirrors 000962 1788265032.610706 read 0x00bd5f thru 0x00d6ce (0x1970 bytes) 000963 1788265032.622447 read 0x020ebe thru 0x02c6b1 (0xb7f4 bytes) 000964 1788265032.645350 read 0x011215 thru 0x013bf8 (0x29e4 bytes) 000965 1788265032.645750 write 0x022632 thru 0x02baaa (0x9479 bytes) 000966 1788265032.700508 trunc from 0x040000 to 0x03565b (0xa9a4 bytes) 000967 1788265032.742411 write 0x03a4a1 thru 0x03f924 (0x5484 bytes) 000969 1788265032.745930 trunc from 0x03cbf6 to 0x03d3fb (0x806 bytes) 000970 1788265032.772040 fallocate 0x01de36 thru 0x02d5bd (0xf788 bytes) 000972 1788265032.779496 punch from 0x004c21 to 0x013bff (0xefdf bytes) 000974 1788265032.786011 mapread 0x01eddf thru 0x022fb2 (0x41d4 bytes) 000975 1788265032.787406 mapwrite 0x01e7ad thru 0x02a00b (0xb85f bytes) 000977 1788265032.810803 trunc from 0x02045f to 0x022608 (0x21aa bytes) 000978 1788265032.816362 write 0x01f600 thru 0x020af4 (0x14f5 bytes) 000979 1788265032.821386 trunc from 0x032942 to 0x03d1d1 (0xa890 bytes) 000980 1788265032.826361 write 0x009cc6 thru 0x010196 (0x64d1 bytes) 000981 1788265032.835119 trunc from 0x032f55 to 0x038e43 (0x5eef bytes) 000982 1788265032.839468 trunc from 0x03f925 to 0x03dbab (0x1d79 bytes) 000983 1788265032.856379 write 0x026f22 thru 0x02fce6 (0x8dc5 bytes) 000984 1788265032.858207 mapread 0x00f94a thru 0x010d73 (0x142a bytes) 000985 1788265032.859513 write 0x017c04 thru 0x023d73 (0xc170 bytes) 000986 1788265032.861495 mapwrite 0x003467 thru 0x009925 (0x64bf bytes) 000987 1788265032.909355 read 0x010e82 thru 0x01ee63 (0xdfe2 bytes) 000988 1788265032.930608 mapwrite 0x0065ab thru 0x010b17 (0xa56d bytes) 000990 1788265032.975761 mapwrite 0x00ceae thru 0x011231 (0x4384 bytes) 000991 1788265033.013232 MIRROR_RESYNC 7 mirrors 000992 1788265035.215885 mapread 0x005a6e thru 0x0065e2 (0xb75 bytes) 000994 1788265035.226917 trunc from 0x0126c3 to 0x01fc7d (0xd5bb bytes) 000995 1788265035.267153 read 0x009623 thru 0x014fe3 (0xb9c1 bytes) 000996 1788265035.310146 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x19b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dbac): Mirror 6: 0xcbd578de (stale) Mirror 7: 0xcbd578de (stale) Mirror 8: 0xcbd578de (stale) Mirror 9: 0xcbd578de (stale) Mirror 10: 0xcbd578de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dbac): Mirror 6: 0xcbd578de (stale) Mirror 7: 0xcbd578de (stale) Mirror 8: 0xcbd578de (stale) Mirror 9: 0xcbd578de (stale) Mirror 10: 0xcbd578de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dbac: skipped 000997 1788265035.809884 write 0x02724d thru 0x030bfc (0x99b0 bytes) 000998 1788265035.830606 trunc from 0x02518a to 0x026145 (0xfbc bytes) 000999 1788265035.838837 write 0x025734 thru 0x0356d2 (0xff9f bytes) 001000 1788265035.849325 write 0x015e3a thru 0x01699d (0xb64 bytes) All operations completed A-OK! PASS 70a (95s) == sanity-flr test 71: check mirror extend parent fid ==== 08:17:20 (1788265040) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1d9:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 08:17:31 (1788265051) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.209314 s, 10.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131657 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 08:17:40 (1788265060) Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre2 Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7769952 .. Extending file size to 9105696 .. Extending file size to 7958144 .. Extending file size to 8972128 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 61368 61369 61370 61373 61376 192.168.203.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg338-client.virtnet /mnt/lustre2 (opts:) 192.168.203.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg338-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:18:59 (1788265139) Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre2 Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8837056 .. Extending file size to 5940672 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7973344 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8738432 .. Extending file size to 9120608 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62915 62916 62919 62920 62921 failed 192.168.203.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg338-client.virtnet /mnt/lustre2 (opts:) 192.168.203.138@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg338-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (75s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:20:14 (1788265214) Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1269032 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2204 1269100 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25700 3581320 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17776 3589244 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43476 7170564 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2272 1269032 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2204 1269100 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25700 3581320 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17776 3589244 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43476 7170564 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173617 s, 12.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.243882 s, 8.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf951eedc (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg338-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:20:35 (1788265235) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0737959 s, 28.4 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 08:20:42 (1788265242) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:20:49 (1788265249) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0835277 s, 25.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x238:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x239:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:20:57 (1788265257) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0989564 s, 21.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x239:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x239:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23a:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23c:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:21:04 (1788265264) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0782399 s, 26.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23e:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:21:36 (1788265296) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg338-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452066 s, 23.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x240:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23c:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x240:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23d:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:22:21 (1788265341) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg338-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0885967 s, 23.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (49s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:23:10 (1788265390) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg338-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg338-server: Pool lustre.test_204f created Adding targets to pool oleg338-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg338-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20233 s, 10.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg338-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg338-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg338-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:24:24 (1788265464) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:24:31 (1788265471) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:24:38 (1788265478) oleg338-server: Pool lustre.test_206 created Adding targets to pool oleg338-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg338-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg338-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg338-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg338-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:25:05 (1788265505) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862903 s, 12.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385995 s, 27.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x512bcf23 Mirror 3: 0x512bcf23 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:25:16 (1788265516) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:25:19 (1788265519) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:25:23 (1788265523) Starting client: oleg338-client.virtnet: -o user_xattr,flock 192.168.203.138@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0656873 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xd00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x950000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x950000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x950000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2440000 Read this (4 bytes): BBBB 192.168.203.138@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg338-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:26:00 (1788265560) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0156828 s, 522 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:26:08 (1788265568) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217648 s, 48.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7f lmm_fid: [0x200000405:0x7f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 08:26:10.000000000 -0400 Modify: 2026-09-01 08:26:10.000000000 -0400 Change: 2026-09-01 08:26:11.000000000 -0400 Birth: 2026-09-01 08:26:10.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:26:15 (1788265575) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517881 s, 20.2 MB/s oleg338-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg338-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg338-server: O/280000401/d25/139: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:26:36 (1788265596) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.195481 s, 210 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0136008 s, 301 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 1 08:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x84:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 1 08:26 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:26:44 (1788265604) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:26:51 (1788265611) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1831 sec =========== 08:26:59 (1788265619) === sanity-flr: start cleanup 08:27:00 (1788265620) === === sanity-flr: finish cleanup 08:27:07 (1788265627) ===