-----============= acceptance-small: sanity-flr ============----- Tue Sep 15 09:16:20 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg232-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:16:42 (1789478202) === oleg232-client.virtnet: executing check_config_client /mnt/lustre oleg232-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg232-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa05109bc3000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa05109bc3000.idle_timeout=debug disable quota as required oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:17:22 (1789478242) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7648] == sanity-flr test 0a: lfs mirror create with -N option == 09:17:25 (1789478245) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:17:43 (1789478263) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:17:47 (1789478267) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:17:51 (1789478271) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:18:20 (1789478300) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:18:23 (1789478303) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 09:18:27 (1789478307) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa05109bc3000.stats=clear osc.lustre-OST0001-osc-ffffa05109bc3000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:18:37 (1789478317) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:18:50 (1789478330) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:19:00 (1789478340) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:19:04 (1789478344) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:19:13 (1789478353) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 09:19:23 (1789478363) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:19:30 (1789478370) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:19:33 (1789478373) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:19:40 (1789478380) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:19:52 (1789478392) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:20:01 (1789478401) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.68085 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218281 s, 4.8 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:20:13 (1789478413) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.258793 s, 36.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:20:22 (1789478422) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:20:33 (1789478433) Stopping /mnt/lustre-ost1 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa05109bc3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa05109bc3000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg232-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg232-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa05109bc3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa05109bc3000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg232-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (140s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:22:52 (1789478572) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.629155 s, 16.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa05109bc3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa05109bc3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg232-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:23:39 (1789478619) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa05109bc3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa05109bc3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg232-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa05109bc3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa05109bc3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg232-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (85s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:25:04 (1789478704) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.38547 s, 7.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 15 09:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.502s user 0m0.001s sys 0m0.210s Stopping /mnt/lustre-ost1 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa05109bc3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa05109bc3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.062s user 0m0.000s sys 0m0.170s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg232-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg232-server oleg232-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa05109bc3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa05109bc3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.901s user 0m0.001s sys 0m0.167s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg232-server: oleg232-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg232-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa05109bc3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa05109bc3000.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:26:49 (1789478809) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:26:53 (1789478813) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:26:56 (1789478816) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:26:59 (1789478819) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 09:27:06 (1789478826) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.8309 s, 15.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (16s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:27:22 (1789478842) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479618 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464356 s, 22.6 MB/s PASS 36b (10s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:27:33 (1789478853) PASS 36c (12s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:27:44 (1789478864) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000967513 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00273412 s, 375 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00146735 s, 698 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138499 s, 739 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 09:27:56 (1789478876) create test files with size 15747621 .. d2ad98840b684ef52b443a719351f4a9 - 5f98465ca03fad37d4c48989ce99cb8a - 5aaa9805fd29cb3f1672d33638f7429f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 09:28:21 (1789478901) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:28:35 (1789478915) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:28:42 (1789478922) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171483 s, 6.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272501 s, 38.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 09:28:52 (1789478932) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19209 s, 10.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.382373 s, 11.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189729 s, 11.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 09:29:08 (1789478948) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 09:29:11 (1789478951) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:29:15 (1789478955) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636956 s, 16.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 09:29:24 (1789478964) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:29:27 (1789478967) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:29:37 (1789478977) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.584238 s, 17.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:29:53 (1789478993) llite.lustre-ffffa05109dbf000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.492245 s, 21.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffffa0510559c000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:30:10 (1789479010) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:30:51 (1789479051) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:31:26 (1789479086) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 09:31:30 (1789479090) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.304431 s, 10.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.149596 s, 21.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb6252527 (nosync,stale) Mirror 2: 0xb6252527 (nosync,stale) ** verify mirror 2 doesn't change original checksum: d1d05f416a192fdd812ac7ee629ad51e - mirror 1 checksum: 3868e5b7d5f246302ce3601bec439a16 - mirror 2 checksum: d1d05f416a192fdd812ac7ee629ad51e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-15T09:31:34-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-15T09:31:34-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d1d05f416a192fdd812ac7ee629ad51e - mirror 1 checksum: 3868e5b7d5f246302ce3601bec439a16 - mirror 2 checksum: 3868e5b7d5f246302ce3601bec439a16 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-15T09:31:34-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-15T09:31:34-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:31:44 (1789479104) Failing mds1 on oleg232-server Stopping /mnt/lustre-mds1 (opts:) on oleg232-server 09:31:51 (1789479111) shut down facet: mds1 facet_host: oleg232-server facet_failover_host: oleg232-server Failover mds1 to oleg232-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 09:32:24 (1789479144) targets are mounted 09:32:24 (1789479144) facet_failover done oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249343 s, 42.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:32:44 (1789479164) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:32:48 (1789479168) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:32:51 (1789479171) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00794647 s, 2.1 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:32:59 (1789479179) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:33:03 (1789479183) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0109671 s, 373 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.115113 s, 35.6 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:33:11 (1789479191) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:33:34 (1789479214) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:34:02 (1789479242) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:34:29 (1789479269) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (28s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:34:57 (1789479297) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 09:35:04 (1789479304) mirror create pid 54381 mirror split pid 54383 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54381 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54383 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 09:36:12 (1789479372) Seed set to 9374 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x20e15 000003 1789479374.782769 trunc from 00000000 to 0x020e14 (0x20e15 bytes) 000004 1789479374.807056 mapwrite 0x009d66 thru 0x0116a7 (0x7942 bytes) 000005 1789479374.959242 read 0x000729 thru 0x00a633 (0x9f0b bytes) 000006 1789479374.973296 read 0x00ebff thru 0x015721 (0x6b23 bytes) 000008 1789479374.992745 read 0x019bb0 thru 0x020e14 (0x7265 bytes) 000010 1789479374.995082 mapread 0x002777 thru 0x00a31c (0x7ba6 bytes) 000011 1789479374.996838 mapread 0x004a78 thru 0x00a244 (0x57cd bytes) 000012 1789479375.003958 MIRROR_EXTEND to 2 mirrors 000013 1789479376.349229 read 0x01d4ff thru 0x020e14 (0x3916 bytes) 000014 1789479376.356831 mapwrite 0x010c1d thru 0x011c24 (0x1008 bytes) 000016 1789479376.555946 mapread 0x01e51a thru 0x020e14 (0x28fb bytes) 000017 1789479376.580102 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) truncating to largest ever: 0x3aed9 000018 1789479376.728231 trunc from 0x020e15 to 0x03aed8 (0x1a0c4 bytes) 000020 1789479376.761960 mapread 0x009aa8 thru 0x01815b (0xe6b4 bytes) 000021 1789479376.797746 mapread 0x005921 thru 0x00e20b (0x88eb bytes) 000022 1789479376.798987 mapwrite 0x03fbd5 thru 0x03ffff (0x42b bytes) 000024 1789479376.903814 MIRROR_EXTEND to 2 mirrors 000025 1789479377.873098 mapread 0x02b39e thru 0x036bce (0xb831 bytes) 000027 1789479377.932261 write 0x0079dd thru 0x007e77 (0x49b bytes) 000028 1789479378.041915 write 0x02979d thru 0x0347d8 (0xb03c bytes) 000030 1789479378.088259 write 0x01c62b thru 0x026888 (0xa25e bytes) 000031 1789479378.105823 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf9c956ff (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xce845735 Mirror 3: 0xce845735 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000032 1789479379.068751 mapread 0x01f5d0 thru 0x0297e9 (0xa21a bytes) 000034 1789479379.102673 trunc from 0x040000 to 0x037d85 (0x827a bytes) 000036 1789479379.158887 trunc from 0x037d86 to 0x004e10 (0x32f75 bytes) 000037 1789479379.177531 read 0x000ce0 thru 0x004e10 (0x4131 bytes) 000038 1789479379.185781 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4e11): Mirror 3: 0xb2ec90a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4e11): Mirror 3: 0xb2ec90a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4e11: skipped 000039 1789479379.477738 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) 000040 1789479379.684329 mapwrite 0x0205f4 thru 0x02fbf0 (0xf5fd bytes) 000046 1789479379.857381 mapread 0x019ed3 thru 0x02705c (0xd18a bytes) 000047 1789479379.883319 mapread 0x01bbda thru 0x01e63d (0x2a64 bytes) 000048 1789479379.887744 mapwrite 0x00ec26 thru 0x019c1d (0xaff8 bytes) 000050 1789479379.949964 read 0x007dbd thru 0x00ec8f (0x6ed3 bytes) 000051 1789479379.950627 trunc from 0x02fbf1 to 0x01567a (0x1a576 bytes) 000052 1789479379.979991 mapread 0x012966 thru 0x01567a (0x2d15 bytes) 000053 1789479379.980855 mapread 0x014663 thru 0x01567a (0x1018 bytes) 000055 1789479379.981972 mapread 0x003833 thru 0x007c01 (0x43cf bytes) 000056 1789479380.002962 read 0x00c963 thru 0x01567a (0x8d18 bytes) 000058 1789479380.005211 read 0x0139bc thru 0x01567a (0x1cbf bytes) 000064 1789479380.022920 read 0x00468b thru 0x00c58a (0x7f00 bytes) 000068 1789479380.039663 mapwrite 0x028def thru 0x034abc (0xbcce bytes) 000069 1789479380.184399 write 0x00279e thru 0x00c345 (0x9ba8 bytes) 000070 1789479380.185687 write 0x03102c thru 0x032b88 (0x1b5d bytes) 000071 1789479380.192871 write 0x00ce03 thru 0x0184b4 (0xb6b2 bytes) 000075 1789479380.210830 write 0x02a32f thru 0x02ad16 (0x9e8 bytes) 000077 1789479380.213195 mapwrite 0x00bbc9 thru 0x0171a4 (0xb5dc bytes) 000079 1789479380.324134 trunc from 0x034abd to 0x00de67 (0x26c55 bytes) 000080 1789479380.360538 trunc from 0x00de68 to 0x014fd3 (0x716c bytes) 000083 1789479380.402890 read 0x004bd3 thru 0x00be2d (0x725b bytes) 000086 1789479380.404335 write 0x037d03 thru 0x03b747 (0x3a45 bytes) 000087 1789479380.421407 mapread 0x02c7d2 thru 0x03873c (0xbf6b bytes) 000088 1789479380.438472 trunc from 0x03b748 to 0x012851 (0x28ef6 bytes) 000090 1789479380.466118 write 0x0151ff thru 0x01761f (0x2421 bytes) 000092 1789479380.468415 read 0x013d3a thru 0x014dcd (0x1094 bytes) 000094 1789479380.483035 read 0x005c05 thru 0x006c79 (0x1075 bytes) 000095 1789479380.485239 mapread 0x006e11 thru 0x0115f6 (0xa7e6 bytes) 000096 1789479380.488480 write 0x034f71 thru 0x038886 (0x3916 bytes) 000100 1789479380.496255 trunc from 0x038887 to 0x00978e (0x2f0f8 bytes) 000103 1789479380.523173 mapwrite 0x01dee6 thru 0x0267f5 (0x8910 bytes) 000106 1789479380.806339 trunc from 0x0267f6 to 0x01b4ed (0xb308 bytes) 000107 1789479380.830313 mapread 0x00a038 thru 0x00d928 (0x38f1 bytes) 000108 1789479380.838730 MIRROR_EXTEND to 2 mirrors 000109 1789479381.931682 mapread 0x000a36 thru 0x00b79e (0xad69 bytes) 000111 1789479381.964901 read 0x01ada0 thru 0x01b4ed (0x74e bytes) 000112 1789479381.969725 mapread 0x01039c thru 0x017af5 (0x775a bytes) 000113 1789479381.981868 read 0x00ac96 thru 0x0196b6 (0xea21 bytes) 000114 1789479381.985889 MIRROR_RESYNC 2 mirrors 000116 1789479382.276601 read 0x0145bd thru 0x01b4ed (0x6f31 bytes) 000117 1789479382.276892 trunc from 0x01b4ee to 0x0307d8 (0x152eb bytes) 000119 1789479382.332428 mapwrite 0x01c6d5 thru 0x028460 (0xbd8c bytes) 000120 1789479382.383574 mapwrite 0x01d30a thru 0x0271a3 (0x9e9a bytes) 000121 1789479382.418000 mapwrite 0x033d9b thru 0x03ef08 (0xb16e bytes) 000123 1789479382.498953 MIRROR_RESYNC 2 mirrors 000124 1789479383.205025 MIRROR_EXTEND to 3 mirrors 000125 1789479383.977908 MIRROR_RESYNC 3 mirrors 000126 1789479384.449176 write 0x02d0bb thru 0x02f58f (0x24d5 bytes) 000128 1789479384.583857 read 0x00abbb thru 0x015aa7 (0xaeed bytes) 000129 1789479384.601216 read 0x00d7a8 thru 0x01577b (0x7fd4 bytes) 000130 1789479384.607084 read 0x02c3db thru 0x0392f2 (0xcf18 bytes) 000131 1789479384.636065 trunc from 0x03ef09 to 0x019562 (0x259a6 bytes) 000133 1789479384.679716 read 0x016f88 thru 0x019562 (0x25db bytes) 000134 1789479384.681712 trunc from 0x019563 to 0x02521e (0xbcbc bytes) 000135 1789479384.698073 write 0x021bc7 thru 0x02cc92 (0xb0cc bytes) 000136 1789479384.717160 write 0x0100ba thru 0x01a693 (0xa5da bytes) 000137 1789479384.720047 mapwrite 0x024a31 thru 0x0287bb (0x3d8b bytes) 000143 1789479384.770043 trunc from 0x02cc93 to 0x0063df (0x268b3 bytes) 000145 1789479384.787342 read 0x004760 thru 0x0063df (0x1c80 bytes) 000146 1789479384.787920 mapread 0x001275 thru 0x0063df (0x516b bytes) 000148 1789479384.789154 mapwrite 0x0378b1 thru 0x0396de (0x1e2e bytes) 000152 1789479384.881254 mapwrite 0x021597 thru 0x02945d (0x7ec7 bytes) 000153 1789479384.957975 read 0x01437b thru 0x01a5ee (0x6274 bytes) 000156 1789479384.971808 trunc from 0x0396df to 0x023330 (0x163ae bytes) 000158 1789479385.033389 read 0x0152c8 thru 0x0184a1 (0x31da bytes) 000161 1789479385.042809 mapwrite 0x0091a5 thru 0x01338d (0xa1e9 bytes) 000162 1789479385.150137 read 0x0138e6 thru 0x01f4f3 (0xbc0e bytes) 000166 1789479385.159312 MIRROR_RESYNC 3 mirrors 000169 1789479385.862863 write 0x0090cc thru 0x01825e (0xf193 bytes) 000170 1789479385.961892 read 0x01be29 thru 0x023330 (0x7508 bytes) 000173 1789479385.975327 mapread 0x01d157 thru 0x023330 (0x61da bytes) 000174 1789479385.976088 write 0x022793 thru 0x024823 (0x2091 bytes) 000175 1789479385.977807 mapread 0x004057 thru 0x010e4d (0xcdf7 bytes) 000176 1789479386.016823 mapread 0x00c0bb thru 0x01b2cd (0xf213 bytes) 000178 1789479386.049131 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24824): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24824): Mirror 2: 0x1324caad (stale) Mirror 3: 0x1324caad (stale) CRC-32 checksum value for chunk [0, 0x24824): Mirror 1: 0x9e2088ed Mirror 4: 0x9e2088ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24824: skipped 000179 1789479387.079125 trunc from 0x024824 to 0x01f58f (0x5294 bytes) 000180 1789479387.140710 read 0x01e831 thru 0x01f58f (0xd5f bytes) 000181 1789479387.146940 write 0x00bad4 thru 0x012d1c (0x7249 bytes) 000183 1789479387.162147 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f590): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f590): Mirror 2: 0xfbe90f13 (stale) Mirror 3: 0xfbe90f13 (stale) Mirror 4: 0x6e56b8d9 (stale) CRC-32 checksum value for chunk [0, 0x1f590): Mirror 1: 0x6abd1603 Mirror 5: 0x6abd1603 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f590: skipped 000184 1789479388.161603 mapread 0x0051b8 thru 0x007174 (0x1fbd bytes) 000185 1789479388.185189 write 0x034dde thru 0x03ffff (0xb222 bytes) 000186 1789479388.248092 mapwrite 0x01b69a thru 0x02a802 (0xf169 bytes) 000187 1789479388.329173 write 0x02430d thru 0x032780 (0xe474 bytes) 000188 1789479388.334585 mapwrite 0x0259fc thru 0x033023 (0xd628 bytes) 000189 1789479388.376853 write 0x0016cb thru 0x001a18 (0x34e bytes) 000190 1789479388.389087 mapread 0x026ecc thru 0x0290cb (0x2200 bytes) 000191 1789479388.390233 mapread 0x032ec2 thru 0x034732 (0x1871 bytes) 000192 1789479388.391540 mapread 0x024a57 thru 0x02fb0c (0xb0b6 bytes) 000193 1789479388.392838 mapread 0x0357a4 thru 0x03ffff (0xa85c bytes) 000196 1789479388.395050 mapwrite 0x039968 thru 0x03ffff (0x6698 bytes) 000197 1789479388.439035 trunc from 0x040000 to 0x00e5eb (0x31a14 bytes) 000199 1789479388.540847 read 0x00b8ef thru 0x00e5eb (0x2cfd bytes) 000200 1789479388.545038 mapwrite 0x03fe53 thru 0x03ffff (0x1ad bytes) 000202 1789479388.712959 MIRROR_RESYNC 5 mirrors 000203 1789479390.270424 mapwrite 0x0092bb thru 0x00a773 (0x14b9 bytes) 000204 1789479390.432177 mapwrite 0x025043 thru 0x02d556 (0x8514 bytes) 000205 1789479390.517727 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5affeac5 (stale) Mirror 3: 0x5affeac5 (stale) Mirror 4: 0x5affeac5 (stale) Mirror 5: 0x5affeac5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5a858099 Mirror 6: 0x5a858099 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000206 1789479391.865786 mapwrite 0x03b4dd thru 0x03ffff (0x4b23 bytes) 000208 1789479391.996083 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5affeac5 (stale) Mirror 3: 0x5affeac5 (stale) Mirror 4: 0x5affeac5 (stale) Mirror 5: 0x5affeac5 (stale) Mirror 6: 0x5a858099 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5edba42e Mirror 7: 0x5edba42e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000212 1789479392.810628 write 0x00adbe thru 0x00be67 (0x10aa bytes) 000213 1789479392.879905 mapwrite 0x029700 thru 0x038330 (0xec31 bytes) 000215 1789479392.944949 trunc from 0x040000 to 0x02404a (0x1bfb5 bytes) 000217 1789479392.969713 mapwrite 0x02d4b8 thru 0x032723 (0x526c bytes) 000218 1789479393.045615 read 0x00ecc9 thru 0x0187b9 (0x9af1 bytes) 000219 1789479393.062215 read 0x000bf5 thru 0x008548 (0x7954 bytes) 000220 1789479393.070825 mapwrite 0x00c001 thru 0x00d75b (0x175b bytes) 000222 1789479393.104481 mapread 0x013368 thru 0x021202 (0xde9b bytes) 000223 1789479393.105302 trunc from 0x032724 to 0x013613 (0x1f110 bytes) 000225 1789479393.121126 MIRROR_RESYNC 7 mirrors 000227 1789479394.464191 trunc from 0x013614 to 0x006d51 (0xc8c2 bytes) 000228 1789479394.653077 read 0x00404f thru 0x006d51 (0x2d03 bytes) 000230 1789479394.662118 read 0x0048c8 thru 0x006d51 (0x248a bytes) 000231 1789479394.664163 write 0x03c573 thru 0x03ee26 (0x28b4 bytes) 000232 1789479394.671163 write 0x024b1d thru 0x0319c7 (0xceab bytes) 000233 1789479394.699598 read 0x004c6d thru 0x00dfb4 (0x9348 bytes) 000234 1789479394.703028 write 0x003d1b thru 0x00a3b4 (0x669a bytes) 000235 1789479394.714057 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ee27): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee27): Mirror 2: 0x3fcc1ad5 (stale) Mirror 3: 0x3fcc1ad5 (stale) Mirror 4: 0x3fcc1ad5 (stale) Mirror 5: 0x3fcc1ad5 (stale) Mirror 6: 0x3fcc1ad5 (stale) Mirror 7: 0x3fcc1ad5 (stale) CRC-32 checksum value for chunk [0, 0x3ee27): Mirror 1: 0xe36922a0 Mirror 8: 0xe36922a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee27: skipped 000237 1789479396.344115 write 0x01fb68 thru 0x023250 (0x36e9 bytes) 000240 1789479396.465368 mapwrite 0x001a52 thru 0x01036e (0xe91d bytes) 000241 1789479396.615418 read 0x03a7df thru 0x03ee26 (0x4648 bytes) 000242 1789479396.630417 trunc from 0x03ee27 to 0x02df4a (0x10edc bytes) 000243 1789479396.665478 read 0x022fce thru 0x02b01b (0x804e bytes) 000247 1789479396.685662 mapwrite 0x033b3f thru 0x0393a6 (0x5868 bytes) 000248 1789479396.801409 mapwrite 0x01597b thru 0x0212c8 (0xb94e bytes) 000250 1789479396.848815 MIRROR_RESYNC 8 mirrors 000251 1789479398.546609 trunc from 0x0393a7 to 0x0347a9 (0x4bfd bytes) 000252 1789479398.617653 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x347aa): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x347aa): Mirror 2: 0xaffb6849 (stale) Mirror 3: 0xaffb6849 (stale) Mirror 4: 0xaffb6849 (stale) Mirror 5: 0xaffb6849 (stale) Mirror 6: 0xaffb6849 (stale) Mirror 7: 0xaffb6849 (stale) Mirror 8: 0xaffb6849 (stale) CRC-32 checksum value for chunk [0, 0x347aa): Mirror 1: 0xaffb6849 Mirror 9: 0xaffb6849 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x347aa: skipped 000254 1789479399.518404 write 0x035580 thru 0x03ffff (0xaa80 bytes) 000256 1789479399.603306 read 0x00449d thru 0x00d0dc (0x8c40 bytes) 000257 1789479399.613078 mapread 0x02f2f5 thru 0x03d627 (0xe333 bytes) 000258 1789479399.634975 mapwrite 0x013a93 thru 0x01adb6 (0x7324 bytes) 000259 1789479399.729773 mapwrite 0x0018ab thru 0x00f859 (0xdfaf bytes) 000261 1789479399.828041 read 0x01817e thru 0x0221ce (0xa051 bytes) 000262 1789479399.839337 read 0x01979c thru 0x020dba (0x761f bytes) 000263 1789479399.840584 read 0x01c3f8 thru 0x024913 (0x851c bytes) 000264 1789479399.841328 mapread 0x02909a thru 0x036d45 (0xdcac bytes) 000267 1789479399.844111 trunc from 0x040000 to 0x01df3b (0x220c4 bytes) 000273 1789479399.906226 mapwrite 0x0130b0 thru 0x0176f3 (0x4644 bytes) 000275 1789479400.011007 write 0x0170d4 thru 0x01c1fa (0x5127 bytes) 000276 1789479400.012598 read 0x002837 thru 0x0049ea (0x21b4 bytes) 000277 1789479400.026790 read 0x01a52e thru 0x01df3b (0x3a0e bytes) 000280 1789479400.029386 mapwrite 0x00fea7 thru 0x01e52e (0xe688 bytes) 000281 1789479400.156979 read 0x01e074 thru 0x01e52e (0x4bb bytes) 000282 1789479400.164980 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e52f): Mirror 2: 0x60494f5a (stale) Mirror 4: 0x60494f5a (stale) Mirror 5: 0x60494f5a (stale) Mirror 6: 0x60494f5a (stale) Mirror 7: 0x60494f5a (stale) Mirror 8: 0x60494f5a (stale) Mirror 9: 0x60494f5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e52f): Mirror 2: 0x60494f5a (stale) Mirror 4: 0x60494f5a (stale) Mirror 5: 0x60494f5a (stale) Mirror 6: 0x60494f5a (stale) Mirror 7: 0x60494f5a (stale) Mirror 8: 0x60494f5a (stale) Mirror 9: 0x60494f5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e52f: skipped 000284 1789479400.704109 read 0x000627 thru 0x00a817 (0xa1f1 bytes) 000287 1789479400.728347 mapwrite 0x03d688 thru 0x03ffff (0x2978 bytes) 000289 1789479400.824071 read 0x0089f8 thru 0x0188f4 (0xfefd bytes) 000290 1789479400.844702 write 0x00b6e9 thru 0x018996 (0xd2ae bytes) 000291 1789479400.848379 write 0x022f00 thru 0x031305 (0xe406 bytes) 000292 1789479400.851681 mapread 0x035155 thru 0x0398a8 (0x4754 bytes) 000293 1789479400.853600 mapwrite 0x01ffe3 thru 0x02df57 (0xdf75 bytes) 000296 1789479400.913570 mapwrite 0x03744f thru 0x03ffff (0x8bb1 bytes) 000297 1789479400.986000 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf3fc9876 (stale) Mirror 4: 0xf3fc9876 (stale) Mirror 5: 0xf3fc9876 (stale) Mirror 6: 0xf3fc9876 (stale) Mirror 7: 0xf3fc9876 (stale) Mirror 8: 0xf3fc9876 (stale) Mirror 9: 0xdafc5b1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x91f6cefa Mirror 10: 0x91f6cefa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000298 1789479402.023135 trunc from 0x040000 to 0x0382b4 (0x7d4b bytes) 000300 1789479402.107773 mapread 0x023585 thru 0x025fe3 (0x2a5f bytes) 000302 1789479402.122683 write 0x0025b5 thru 0x010520 (0xdf6c bytes) 000303 1789479402.142977 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x382b5): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x382b5): Mirror 2: 0x345197b2 (stale) Mirror 4: 0x345197b2 (stale) Mirror 5: 0x345197b2 (stale) Mirror 6: 0x345197b2 (stale) Mirror 7: 0x345197b2 (stale) Mirror 8: 0x345197b2 (stale) Mirror 9: 0x9712e034 (stale) Mirror 10: 0x6338c269 (stale) CRC-32 checksum value for chunk [0, 0x382b5): Mirror 1: 0xe49e757f Mirror 11: 0xe49e757f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x382b5: skipped 000309 1789479403.731336 read 0x02124f thru 0x02c08a (0xae3c bytes) 000310 1789479403.762953 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x382b5): Mirror 2: 0x345197b2 (stale) Mirror 4: 0x345197b2 (stale) Mirror 6: 0x345197b2 (stale) Mirror 7: 0x345197b2 (stale) Mirror 8: 0x345197b2 (stale) Mirror 9: 0x9712e034 (stale) Mirror 10: 0x6338c269 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x382b5): Mirror 2: 0x345197b2 (stale) Mirror 4: 0x345197b2 (stale) Mirror 6: 0x345197b2 (stale) Mirror 7: 0x345197b2 (stale) Mirror 8: 0x345197b2 (stale) Mirror 9: 0x9712e034 (stale) Mirror 10: 0x6338c269 (stale) CRC-32 checksum value for chunk [0, 0x382b5): Mirror 1: 0xe49e757f Mirror 11: 0xe49e757f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x382b5: skipped 000311 1789479404.808402 mapwrite 0x03e8bd thru 0x03ffff (0x1743 bytes) 000312 1789479405.111505 MIRROR_RESYNC 9 mirrors 000314 1789479406.674686 read 0x012850 thru 0x017463 (0x4c14 bytes) 000316 1789479406.688614 mapwrite 0x01ec28 thru 0x02a8ae (0xbc87 bytes) 000317 1789479407.060974 mapread 0x004314 thru 0x00c822 (0x850f bytes) 000318 1789479407.084483 mapwrite 0x0268c7 thru 0x02d37f (0x6ab9 bytes) 000319 1789479407.203297 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9ef7c2c (stale) Mirror 4: 0x9ef7c2c (stale) Mirror 6: 0x9ef7c2c (stale) Mirror 7: 0x9ef7c2c (stale) Mirror 8: 0x9ef7c2c (stale) Mirror 9: 0x9ef7c2c (stale) Mirror 10: 0x9ef7c2c (stale) Mirror 11: 0x9ef7c2c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeebd8095 Mirror 12: 0xeebd8095 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000321 1789479408.530859 mapwrite 0x013fc9 thru 0x023f9e (0xffd6 bytes) 000323 1789479408.828816 read 0x02b09e thru 0x037ce6 (0xcc49 bytes) 000326 1789479408.846435 mapread 0x03b4b5 thru 0x03cf4c (0x1a98 bytes) 000328 1789479408.871630 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9ef7c2c (stale) Mirror 4: 0x9ef7c2c (stale) Mirror 6: 0x9ef7c2c (stale) Mirror 7: 0x9ef7c2c (stale) Mirror 8: 0x9ef7c2c (stale) Mirror 9: 0x9ef7c2c (stale) Mirror 10: 0x9ef7c2c (stale) Mirror 11: 0x9ef7c2c (stale) Mirror 12: 0xeebd8095 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb218cc6c Mirror 13: 0xb218cc6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000329 1789479410.394937 write 0x01f764 thru 0x029f44 (0xa7e1 bytes) 000330 1789479410.464516 mapwrite 0x0036b0 thru 0x013446 (0xfd97 bytes) 000332 1789479410.512753 write 0x035f7c thru 0x038b17 (0x2b9c bytes) 000333 1789479410.517960 mapwrite 0x01a0a3 thru 0x01f24e (0x51ac bytes) 000334 1789479410.553983 read 0x02d094 thru 0x0392ad (0xc21a bytes) 000335 1789479410.564579 write 0x015309 thru 0x02449c (0xf194 bytes) 000336 1789479410.569296 mapwrite 0x03f7e4 thru 0x03ffff (0x81c bytes) 000337 1789479410.633656 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9ef7c2c (stale) Mirror 4: 0x9ef7c2c (stale) Mirror 6: 0x9ef7c2c (stale) Mirror 7: 0x9ef7c2c (stale) Mirror 8: 0x9ef7c2c (stale) Mirror 9: 0x9ef7c2c (stale) Mirror 10: 0x9ef7c2c (stale) Mirror 11: 0x9ef7c2c (stale) Mirror 12: 0xeebd8095 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9ef7c2c (stale) Mirror 4: 0x9ef7c2c (stale) Mirror 6: 0x9ef7c2c (stale) Mirror 7: 0x9ef7c2c (stale) Mirror 8: 0x9ef7c2c (stale) Mirror 9: 0x9ef7c2c (stale) Mirror 10: 0x9ef7c2c (stale) Mirror 11: 0x9ef7c2c (stale) Mirror 12: 0xeebd8095 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000338 1789479411.059522 mapwrite 0x013657 thru 0x016c55 (0x35ff bytes) 000339 1789479411.099499 write 0x03e778 thru 0x03ffff (0x1888 bytes) 000340 1789479411.102901 mapread 0x03b25e thru 0x03ffff (0x4da2 bytes) 000341 1789479411.111031 MIRROR_RESYNC 10 mirrors 000343 1789479412.148675 trunc from 0x040000 to 0x012f5e (0x2d0a1 bytes) truncating to largest ever: 0x3f88a 000344 1789479412.246493 trunc from 0x012f5f to 0x03f889 (0x2c92b bytes) 000345 1789479412.265726 mapread 0x02c002 thru 0x034f49 (0x8f48 bytes) 000348 1789479412.281795 mapread 0x021e87 thru 0x03001e (0xe198 bytes) 000349 1789479412.289726 write 0x016734 thru 0x01c8ef (0x61bc bytes) 000350 1789479412.293975 read 0x00dab1 thru 0x013fc3 (0x6513 bytes) 000353 1789479412.301813 read 0x01b0cc thru 0x01f22e (0x4163 bytes) 000354 1789479412.305622 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f88a): Mirror 2: 0x83f649d5 (stale) Mirror 4: 0x83f649d5 (stale) Mirror 7: 0x83f649d5 (stale) Mirror 8: 0x83f649d5 (stale) Mirror 9: 0x83f649d5 (stale) Mirror 10: 0x83f649d5 (stale) Mirror 11: 0x83f649d5 (stale) Mirror 12: 0x83f649d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f88a): Mirror 2: 0x83f649d5 (stale) Mirror 4: 0x83f649d5 (stale) Mirror 7: 0x83f649d5 (stale) Mirror 8: 0x83f649d5 (stale) Mirror 9: 0x83f649d5 (stale) Mirror 10: 0x83f649d5 (stale) Mirror 11: 0x83f649d5 (stale) Mirror 12: 0x83f649d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f88a: skipped 000355 1789479412.911545 mapread 0x01c0a5 thru 0x01dac6 (0x1a22 bytes) 000356 1789479412.922655 mapwrite 0x0115d6 thru 0x01d3e9 (0xbe14 bytes) 000357 1789479412.979542 mapread 0x003bdf thru 0x00685a (0x2c7c bytes) 000358 1789479412.992139 write 0x019816 thru 0x01cdea (0x35d5 bytes) 000360 1789479412.993175 trunc from 0x03f88a to 0x0210ed (0x1e79c bytes) 000361 1789479413.004559 write 0x02d6ba thru 0x030ef3 (0x383a bytes) 000362 1789479413.005926 trunc from 0x030ef4 to 0x018ba4 (0x1834f bytes) 000364 1789479413.022622 MIRROR_RESYNC 9 mirrors 000366 1789479414.932484 write 0x01ff4a thru 0x02dd09 (0xddc0 bytes) 000367 1789479415.098987 mapread 0x019b28 thru 0x024788 (0xac61 bytes) 000368 1789479415.160688 read 0x0122a9 thru 0x01b53e (0x9296 bytes) 000369 1789479415.173681 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2dd0a): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd0a): Mirror 2: 0xa20db50 (stale) Mirror 4: 0xa20db50 (stale) Mirror 7: 0xa20db50 (stale) Mirror 8: 0xa20db50 (stale) Mirror 9: 0xa20db50 (stale) Mirror 10: 0xa20db50 (stale) Mirror 11: 0xa20db50 (stale) Mirror 12: 0xa20db50 (stale) CRC-32 checksum value for chunk [0, 0x2dd0a): Mirror 1: 0x6ab57536 Mirror 13: 0x6ab57536 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd0a: skipped 000372 1789479416.326413 mapwrite 0x025333 thru 0x02d567 (0x8235 bytes) 000374 1789479416.487510 trunc from 0x02dd0a to 0x038172 (0xa469 bytes) 000375 1789479416.516558 write 0x03a132 thru 0x03ffff (0x5ece bytes) 000376 1789479416.517718 mapread 0x03dedb thru 0x03e4b7 (0x5dd bytes) 000382 1789479416.522928 MIRROR_RESYNC 10 mirrors 000383 1789479417.687842 mapread 0x032c5c thru 0x03bf84 (0x9329 bytes) 000385 1789479417.707729 read 0x02118d thru 0x023646 (0x24ba bytes) 000386 1789479417.718210 MIRROR_EXTEND to 11 mirrors 000387 1789479418.179933 trunc from 0x040000 to 0x001355 (0x3ecaa bytes) 000389 1789479418.228318 mapwrite 0x006fb8 thru 0x008afd (0x1b46 bytes) 000391 1789479418.278960 write 0x026e83 thru 0x034e4f (0xdfcd bytes) 000393 1789479418.280557 trunc from 0x034e50 to 0x01d744 (0x1770b bytes) 000396 1789479418.292889 trunc from 0x01d745 to 0x01708d (0x66b7 bytes) 000399 1789479418.304419 mapwrite 0x03ec93 thru 0x03ffff (0x136d bytes) 000404 1789479418.345912 read 0x038901 thru 0x03ffff (0x76ff bytes) 000405 1789479418.346511 write 0x029309 thru 0x02b8c1 (0x25b9 bytes) 000406 1789479418.352382 write 0x02acce thru 0x03a9a2 (0xfcd5 bytes) 000407 1789479418.357138 MIRROR_RESYNC 11 mirrors 000408 1789479420.965834 write 0x00d256 thru 0x0185c6 (0xb371 bytes) 000409 1789479421.016499 trunc from 0x040000 to 0x0142a5 (0x2bd5a bytes) 000410 1789479421.057761 write 0x00a5bb thru 0x00cc0a (0x2650 bytes) 000411 1789479421.085810 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x142a6): Mirror 2: 0xd2c84d04 (stale) Mirror 7: 0xd2c84d04 (stale) Mirror 8: 0xd2c84d04 (stale) Mirror 9: 0xd2c84d04 (stale) Mirror 10: 0xd2c84d04 (stale) Mirror 11: 0xd2c84d04 (stale) Mirror 12: 0xd2c84d04 (stale) Mirror 13: 0xd2c84d04 (stale) Mirror 14: 0xd2c84d04 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x142a6): Mirror 2: 0xd2c84d04 (stale) Mirror 7: 0xd2c84d04 (stale) Mirror 8: 0xd2c84d04 (stale) Mirror 9: 0xd2c84d04 (stale) Mirror 10: 0xd2c84d04 (stale) Mirror 11: 0xd2c84d04 (stale) Mirror 12: 0xd2c84d04 (stale) Mirror 13: 0xd2c84d04 (stale) Mirror 14: 0xd2c84d04 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x142a6: skipped 000412 1789479421.676775 write 0x038e42 thru 0x03ffff (0x71be bytes) 000413 1789479421.677831 trunc from 0x040000 to 0x005d75 (0x3a28a bytes) 000414 1789479421.689769 mapwrite 0x00fd70 thru 0x014ce5 (0x4f76 bytes) 000415 1789479421.805044 trunc from 0x014ce6 to 0x025fba (0x112d5 bytes) 000417 1789479421.829029 mapwrite 0x024522 thru 0x03142c (0xcf0b bytes) 000422 1789479422.035995 read 0x01f23b thru 0x02cf82 (0xdd48 bytes) 000425 1789479422.045925 write 0x00e915 thru 0x01a30e (0xb9fa bytes) 000426 1789479422.060237 write 0x03ed86 thru 0x03ffff (0x127a bytes) 000427 1789479422.065848 mapwrite 0x036446 thru 0x03ffff (0x9bba bytes) 000428 1789479422.175855 trunc from 0x040000 to 0x036b14 (0x94eb bytes) 000429 1789479422.230673 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36b15): Mirror 2: 0x67a4822 (stale) Mirror 8: 0x67a4822 (stale) Mirror 9: 0x67a4822 (stale) Mirror 10: 0x67a4822 (stale) Mirror 11: 0x67a4822 (stale) Mirror 12: 0x67a4822 (stale) Mirror 13: 0x67a4822 (stale) Mirror 14: 0x67a4822 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b15): Mirror 2: 0x67a4822 (stale) Mirror 8: 0x67a4822 (stale) Mirror 9: 0x67a4822 (stale) Mirror 10: 0x67a4822 (stale) Mirror 11: 0x67a4822 (stale) Mirror 12: 0x67a4822 (stale) Mirror 13: 0x67a4822 (stale) Mirror 14: 0x67a4822 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b15: skipped 000430 1789479422.813772 trunc from 0x036b15 to 0x022ac0 (0x14054 bytes) 000432 1789479422.839472 trunc from 0x022ac1 to 0x0185ed (0xa4d3 bytes) 000433 1789479422.861149 MIRROR_RESYNC 9 mirrors 000434 1789479424.636161 MIRROR_EXTEND to 10 mirrors 000435 1789479425.542676 write 0x03e30e thru 0x03ffff (0x1cf2 bytes) 000436 1789479425.670907 write 0x0229b2 thru 0x02c0a5 (0x96f4 bytes) 000438 1789479425.700623 trunc from 0x040000 to 0x036822 (0x97dd bytes) 000439 1789479425.738246 mapwrite 0x01fd16 thru 0x02b23a (0xb525 bytes) 000446 1789479425.853236 mapwrite 0x013329 thru 0x0216bd (0xe395 bytes) 000448 1789479425.905362 write 0x00a693 thru 0x00eda0 (0x470e bytes) 000449 1789479425.910912 mapwrite 0x01a9b5 thru 0x01ab80 (0x1cc bytes) 000450 1789479426.035805 mapwrite 0x00c671 thru 0x013d12 (0x76a2 bytes) 000451 1789479426.146824 MIRROR_RESYNC 10 mirrors 000453 1789479427.824720 read 0x02ac27 thru 0x036822 (0xbbfc bytes) 000456 1789479427.835864 write 0x00ce17 thru 0x01af2d (0xe117 bytes) 000457 1789479427.914238 read 0x007550 thru 0x015599 (0xe04a bytes) 000460 1789479427.931249 trunc from 0x036823 to 0x037a53 (0x1231 bytes) 000467 1789479427.956815 read 0x006c31 thru 0x00bd26 (0x50f6 bytes) 000471 1789479427.960494 trunc from 0x037a54 to 0x03d656 (0x5c03 bytes) 000472 1789479427.978544 write 0x03dc4f thru 0x03ffff (0x23b1 bytes) 000473 1789479427.986541 trunc from 0x040000 to 0x032092 (0xdf6d bytes) 000474 1789479428.004315 mapread 0x02c649 thru 0x02d1c7 (0xb7f bytes) 000476 1789479428.012613 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x32093): Mirror 2: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32093): Mirror 2: 0x2d4f8f72 (stale) Mirror 8: 0x2d4f8f72 (stale) Mirror 9: 0x2d4f8f72 (stale) Mirror 10: 0x2d4f8f72 (stale) Mirror 11: 0x2d4f8f72 (stale) Mirror 12: 0x2d4f8f72 (stale) Mirror 13: 0x2d4f8f72 (stale) Mirror 14: 0x2d4f8f72 (stale) Mirror 15: 0x2d4f8f72 (stale) CRC-32 checksum value for chunk [0, 0x32093): Mirror 1: 0x4fe7f1e1 Mirror 16: 0x4fe7f1e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32093: skipped 000477 1789479429.316574 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x32093): Mirror 2: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32093): Mirror 2: 0x2d4f8f72 (stale) Mirror 8: 0x2d4f8f72 (stale) Mirror 9: 0x2d4f8f72 (stale) Mirror 10: 0x2d4f8f72 (stale) Mirror 11: 0x2d4f8f72 (stale) Mirror 12: 0x2d4f8f72 (stale) Mirror 13: 0x2d4f8f72 (stale) Mirror 14: 0x2d4f8f72 (stale) Mirror 15: 0x2d4f8f72 (stale) CRC-32 checksum value for chunk [0, 0x32093): Mirror 1: 0x4fe7f1e1 Mirror 16: 0x4fe7f1e1 Mirror 17: 0x4fe7f1e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32093: skipped 000479 1789479430.208514 mapread 0x02978a thru 0x02edc1 (0x5638 bytes) 000480 1789479430.254885 trunc from 0x032093 to 0x014bc5 (0x1d4cd bytes) 000482 1789479430.467415 trunc from 0x014bc6 to 0x034b4a (0x1ff85 bytes) 000484 1789479430.492667 read 0x01e79f thru 0x026fde (0x8840 bytes) 000485 1789479430.499972 mapread 0x0302cb thru 0x034b4a (0x4880 bytes) 000488 1789479430.514636 write 0x0056b9 thru 0x0062e2 (0xc2a bytes) 000489 1789479430.524708 write 0x02024c thru 0x024960 (0x4715 bytes) 000490 1789479430.525969 mapread 0x017685 thru 0x0268e2 (0xf25e bytes) 000491 1789479430.536421 read 0x024fd9 thru 0x032a1e (0xda46 bytes) 000493 1789479430.555146 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34b4b): Mirror 8: 0x490afbc3 (stale) Mirror 9: 0x490afbc3 (stale) Mirror 10: 0x490afbc3 (stale) Mirror 11: 0x490afbc3 (stale) Mirror 12: 0x490afbc3 (stale) Mirror 13: 0x490afbc3 (stale) Mirror 14: 0x490afbc3 (stale) Mirror 15: 0x490afbc3 (stale) Mirror 16: 0xfa25aa77 (stale) Mirror 17: 0xfa25aa77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b4b): Mirror 8: 0x490afbc3 (stale) Mirror 9: 0x490afbc3 (stale) Mirror 10: 0x490afbc3 (stale) Mirror 11: 0x490afbc3 (stale) Mirror 12: 0x490afbc3 (stale) Mirror 13: 0x490afbc3 (stale) Mirror 14: 0x490afbc3 (stale) Mirror 15: 0x490afbc3 (stale) Mirror 16: 0xfa25aa77 (stale) Mirror 17: 0xfa25aa77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b4b: skipped 000494 1789479431.287655 read 0x02621b thru 0x02936d (0x3153 bytes) 000497 1789479431.297291 read 0x03486e thru 0x034b4a (0x2dd bytes) 000498 1789479431.307651 write 0x01b737 thru 0x0224e3 (0x6dad bytes) 000499 1789479431.316543 MIRROR_RESYNC 11 mirrors 000500 1789479434.121172 trunc from 0x034b4b to 0x017e26 (0x1cd24 bytes) 000502 1789479434.328988 mapwrite 0x0006a6 thru 0x001ac7 (0x1422 bytes) 000503 1789479434.436973 mapread 0x0128b8 thru 0x017e26 (0x556f bytes) 000510 1789479434.497659 mapwrite 0x02e5e2 thru 0x037861 (0x9280 bytes) 000511 1789479434.610320 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37862): Mirror 8: 0x78060ac8 (stale) Mirror 9: 0x78060ac8 (stale) Mirror 11: 0x78060ac8 (stale) Mirror 12: 0x78060ac8 (stale) Mirror 13: 0x78060ac8 (stale) Mirror 14: 0x78060ac8 (stale) Mirror 15: 0x78060ac8 (stale) Mirror 16: 0x78060ac8 (stale) Mirror 17: 0x78060ac8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37862): Mirror 8: 0x78060ac8 (stale) Mirror 9: 0x78060ac8 (stale) Mirror 11: 0x78060ac8 (stale) Mirror 12: 0x78060ac8 (stale) Mirror 13: 0x78060ac8 (stale) Mirror 14: 0x78060ac8 (stale) Mirror 15: 0x78060ac8 (stale) Mirror 16: 0x78060ac8 (stale) Mirror 17: 0x78060ac8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37862: skipped 000513 1789479435.236147 mapread 0x01df45 thru 0x027463 (0x951f bytes) 000515 1789479435.258658 read 0x0144d5 thru 0x01c3e9 (0x7f15 bytes) 000518 1789479435.266273 mapwrite 0x021697 thru 0x02a189 (0x8af3 bytes) 000519 1789479435.315237 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37862): Mirror 8: 0x78060ac8 (stale) Mirror 9: 0x78060ac8 (stale) Mirror 11: 0x78060ac8 (stale) Mirror 12: 0x78060ac8 (stale) Mirror 13: 0x78060ac8 (stale) Mirror 14: 0x78060ac8 (stale) Mirror 16: 0x78060ac8 (stale) Mirror 17: 0x78060ac8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37862): Mirror 8: 0x78060ac8 (stale) Mirror 9: 0x78060ac8 (stale) Mirror 11: 0x78060ac8 (stale) Mirror 12: 0x78060ac8 (stale) Mirror 13: 0x78060ac8 (stale) Mirror 14: 0x78060ac8 (stale) Mirror 16: 0x78060ac8 (stale) Mirror 17: 0x78060ac8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37862: skipped 000522 1789479435.932220 read 0x032f12 thru 0x037861 (0x4950 bytes) 000526 1789479435.939536 mapwrite 0x03edee thru 0x03ffff (0x1212 bytes) 000527 1789479435.983996 MIRROR_RESYNC 9 mirrors 000528 1789479436.915529 read 0x0173be thru 0x0201b1 (0x8df4 bytes) 000529 1789479436.923913 write 0x034c03 thru 0x03ffff (0xb3fd bytes) 000530 1789479436.959070 read 0x00cc29 thru 0x010124 (0x34fc bytes) 000531 1789479436.967518 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe9921606 (stale) Mirror 9: 0xe9921606 (stale) Mirror 12: 0xe9921606 (stale) Mirror 13: 0xe9921606 (stale) Mirror 14: 0xe9921606 (stale) Mirror 16: 0xe9921606 (stale) Mirror 17: 0xe9921606 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe9921606 (stale) Mirror 9: 0xe9921606 (stale) Mirror 12: 0xe9921606 (stale) Mirror 13: 0xe9921606 (stale) Mirror 14: 0xe9921606 (stale) Mirror 16: 0xe9921606 (stale) Mirror 17: 0xe9921606 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000532 1789479437.425533 mapread 0x007042 thru 0x010cc0 (0x9c7f bytes) 000534 1789479437.456780 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe9921606 (stale) Mirror 9: 0xe9921606 (stale) Mirror 12: 0xe9921606 (stale) Mirror 13: 0xe9921606 (stale) Mirror 14: 0xe9921606 (stale) Mirror 16: 0xe9921606 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xe9921606 (stale) Mirror 9: 0xe9921606 (stale) Mirror 12: 0xe9921606 (stale) Mirror 13: 0xe9921606 (stale) Mirror 14: 0xe9921606 (stale) Mirror 16: 0xe9921606 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000535 1789479437.958864 read 0x03e20b thru 0x03ffff (0x1df5 bytes) 000536 1789479437.967222 trunc from 0x040000 to 0x03e9a4 (0x165b bytes) 000537 1789479437.982431 write 0x02c266 thru 0x02c29d (0x38 bytes) 000538 1789479437.986566 read 0x02c3a2 thru 0x0316a0 (0x52ff bytes) 000540 1789479437.998295 MIRROR_RESYNC 7 mirrors 000545 1789479439.219671 write 0x001e53 thru 0x00d755 (0xb903 bytes) 000547 1789479439.311088 mapread 0x02f115 thru 0x037f02 (0x8dee bytes) 000548 1789479439.344494 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9a5): Mirror 8: 0x931362fc (stale) Mirror 9: 0x931362fc (stale) Mirror 12: 0x931362fc (stale) Mirror 13: 0x931362fc (stale) Mirror 16: 0x931362fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9a5): Mirror 8: 0x931362fc (stale) Mirror 9: 0x931362fc (stale) Mirror 12: 0x931362fc (stale) Mirror 13: 0x931362fc (stale) Mirror 16: 0x931362fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9a5: skipped 000550 1789479439.928694 mapread 0x01c2c5 thru 0x02c1e7 (0xff23 bytes) 000551 1789479439.969811 mapwrite 0x022f55 thru 0x02fa5e (0xcb0a bytes) 000552 1789479440.065334 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e9a5): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9a5): Mirror 8: 0x931362fc (stale) Mirror 9: 0x931362fc (stale) Mirror 12: 0x931362fc (stale) Mirror 13: 0x931362fc (stale) Mirror 16: 0x931362fc (stale) CRC-32 checksum value for chunk [0, 0x3e9a5): Mirror 1: 0xe772f337 Mirror 17: 0xe772f337 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9a5: skipped 000554 1789479441.000899 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9a5): Mirror 8: 0x931362fc (stale) Mirror 9: 0x931362fc (stale) Mirror 12: 0x931362fc (stale) Mirror 16: 0x931362fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9a5): Mirror 8: 0x931362fc (stale) Mirror 9: 0x931362fc (stale) Mirror 12: 0x931362fc (stale) Mirror 16: 0x931362fc (stale) CRC-32 checksum value for chunk [0, 0x3e9a5): Mirror 1: 0xe772f337 Mirror 17: 0xe772f337 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9a5: skipped 000555 1789479441.731185 mapread 0x0208a5 thru 0x0269f5 (0x6151 bytes) 000556 1789479441.785355 mapread 0x008f62 thru 0x018876 (0xf915 bytes) 000557 1789479441.830555 write 0x00e7d5 thru 0x010ac7 (0x22f3 bytes) 000560 1789479441.950618 read 0x03dc58 thru 0x03e9a4 (0xd4d bytes) 000561 1789479441.959230 write 0x011e70 thru 0x016832 (0x49c3 bytes) 000567 1789479441.981598 write 0x006df2 thru 0x01078a (0x9999 bytes) 000568 1789479441.999627 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9a5): Mirror 8: 0x931362fc (stale) Mirror 9: 0x931362fc (stale) Mirror 16: 0x931362fc (stale) Mirror 17: 0xe772f337 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9a5): Mirror 8: 0x931362fc (stale) Mirror 9: 0x931362fc (stale) Mirror 16: 0x931362fc (stale) Mirror 17: 0xe772f337 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9a5: skipped 000569 1789479442.830703 trunc from 0x03e9a5 to 0x009aa2 (0x34f02 bytes) 000570 1789479442.874061 mapread 0x0044eb thru 0x0090eb (0x4c01 bytes) 000571 1789479442.894523 mapread 0x008ce1 thru 0x009aa2 (0xdc2 bytes) 000573 1789479442.895785 read 0x00178c thru 0x00818c (0x6a01 bytes) 000574 1789479442.896328 read 0x0048ad thru 0x009aa2 (0x51f6 bytes) 000577 1789479442.897780 trunc from 0x009aa3 to 0x025f39 (0x1c497 bytes) 000579 1789479442.910236 write 0x03e979 thru 0x03ffff (0x1687 bytes) 000581 1789479442.911529 write 0x0181f8 thru 0x024ff2 (0xcdfb bytes) 000582 1789479442.917737 read 0x0279e1 thru 0x02a7b4 (0x2dd4 bytes) 000583 1789479442.918206 write 0x019553 thru 0x0196cd (0x17b bytes) 000587 1789479442.919688 trunc from 0x040000 to 0x01b666 (0x24999 bytes) 000588 1789479442.938281 mapread 0x01664a thru 0x01a78f (0x4146 bytes) 000589 1789479442.939210 trunc from 0x01b667 to 0x014d01 (0x6965 bytes) 000590 1789479442.951269 read 0x007829 thru 0x01172c (0x9f04 bytes) 000593 1789479442.952362 mapread 0x004fc7 thru 0x009c0d (0x4c47 bytes) 000594 1789479442.952942 read 0x00c95f thru 0x014d01 (0x83a3 bytes) 000595 1789479442.953479 mapread 0x012059 thru 0x014651 (0x25f9 bytes) 000596 1789479442.954881 read 0x00f78b thru 0x014d01 (0x5577 bytes) 000597 1789479442.955323 read 0x000a19 thru 0x0031c6 (0x27ae bytes) 000598 1789479442.955650 mapread 0x005a7a thru 0x00bbf1 (0x6178 bytes) 000601 1789479442.957333 trunc from 0x014d02 to 0x024c21 (0xff20 bytes) 000604 1789479442.970651 read 0x0084cf thru 0x015ca0 (0xd7d2 bytes) 000605 1789479442.975941 read 0x005206 thru 0x00c0f6 (0x6ef1 bytes) 000609 1789479442.977366 read 0x016562 thru 0x0200ed (0x9b8c bytes) 000611 1789479442.978159 write 0x02d3f8 thru 0x036b5e (0x9767 bytes) 000612 1789479442.980716 mapread 0x00431f thru 0x004e7f (0xb61 bytes) 000613 1789479442.981875 mapread 0x00cc1d thru 0x017927 (0xad0b bytes) 000614 1789479442.986257 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36b5f): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b5f): Mirror 8: 0x7516f897 (stale) Mirror 9: 0x7516f897 (stale) Mirror 16: 0x7516f897 (stale) Mirror 17: 0x95d88db9 (stale) CRC-32 checksum value for chunk [0, 0x36b5f): Mirror 1: 0x7b9f920c Mirror 18: 0x7b9f920c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b5f: skipped 000615 1789479443.926568 MIRROR_RESYNC 6 mirrors 000616 1789479444.745764 mapread 0x034d8c thru 0x036b5e (0x1dd3 bytes) 000617 1789479444.769606 write 0x022e82 thru 0x02ad9f (0x7f1e bytes) 000618 1789479444.831294 mapwrite 0x01e369 thru 0x02d300 (0xef98 bytes) 000619 1789479444.888477 read 0x031dc3 thru 0x036b5e (0x4d9c bytes) 000620 1789479444.893526 write 0x00f3fd thru 0x00fc6e (0x872 bytes) 000621 1789479444.898813 write 0x02a1f3 thru 0x0302e4 (0x60f2 bytes) 000622 1789479444.900393 read 0x0054fc thru 0x00c06b (0x6b70 bytes) 000623 1789479444.907758 read 0x033388 thru 0x036b5e (0x37d7 bytes) 000624 1789479444.908724 mapwrite 0x018aa4 thru 0x01efb9 (0x6516 bytes) 000625 1789479444.947622 write 0x03736f thru 0x03c576 (0x5208 bytes) 000626 1789479444.949726 trunc from 0x03c577 to 0x025fc9 (0x165ad bytes) 000627 1789479444.974605 read 0x0003c7 thru 0x00b404 (0xb03e bytes) 000628 1789479444.975535 mapwrite 0x032f24 thru 0x033087 (0x164 bytes) 000630 1789479445.036305 mapwrite 0x02aacd thru 0x02bf4d (0x1481 bytes) 000631 1789479445.070157 trunc from 0x033088 to 0x032550 (0xb37 bytes) 000632 1789479445.086181 write 0x01389d thru 0x01c6db (0x8e3f bytes) 000633 1789479445.088350 trunc from 0x032551 to 0x0348e7 (0x2397 bytes) 000634 1789479445.102433 trunc from 0x0348e8 to 0x03c70a (0x7e23 bytes) 000635 1789479445.120246 mapread 0x01c177 thru 0x01d018 (0xea2 bytes) 000639 1789479445.123239 mapread 0x03181a thru 0x03c70a (0xaef1 bytes) 000640 1789479445.129944 mapwrite 0x012baa thru 0x01fc65 (0xd0bc bytes) 000641 1789479445.169241 write 0x02937b thru 0x02d505 (0x418b bytes) 000643 1789479445.175518 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c70b): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c70b): Mirror 8: 0xae4e801e (stale) Mirror 9: 0xae4e801e (stale) Mirror 16: 0xae4e801e (stale) Mirror 17: 0xae4e801e (stale) Mirror 18: 0xae4e801e (stale) CRC-32 checksum value for chunk [0, 0x3c70b): Mirror 1: 0xa2a5852 Mirror 19: 0xa2a5852 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c70b: skipped 000645 1789479445.852936 read 0x00ff0b thru 0x01cbef (0xcce5 bytes) 000647 1789479445.875631 trunc from 0x03c70b to 0x0328ec (0x9e1e bytes) 000648 1789479445.985823 trunc from 0x0328ed to 0x024dbd (0xdb2f bytes) 000651 1789479446.021354 mapread 0x0013ba thru 0x00e6ae (0xd2f5 bytes) 000655 1789479446.041420 trunc from 0x024dbe to 0x030d3e (0xbf81 bytes) 000657 1789479446.060484 mapwrite 0x027a3a thru 0x034ba6 (0xd16d bytes) 000659 1789479446.244306 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34ba7): Mirror 9: 0xfd4bfd4a (stale) Mirror 16: 0xfd4bfd4a (stale) Mirror 17: 0xfd4bfd4a (stale) Mirror 18: 0xfd4bfd4a (stale) Mirror 19: 0x5a6a9e06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ba7): Mirror 9: 0xfd4bfd4a (stale) Mirror 16: 0xfd4bfd4a (stale) Mirror 17: 0xfd4bfd4a (stale) Mirror 18: 0xfd4bfd4a (stale) Mirror 19: 0x5a6a9e06 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ba7: skipped 000660 1789479446.643362 trunc from 0x034ba7 to 0x01c62d (0x18579 bytes) 000662 1789479446.671305 mapread 0x0176e1 thru 0x01c62d (0x4f4d bytes) 000667 1789479446.703135 trunc from 0x01c62e to 0x0301ba (0x13b8d bytes) 000668 1789479446.726675 read 0x006aea thru 0x010fae (0xa4c5 bytes) 000669 1789479446.744071 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x301bb): Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301bb): Mirror 9: 0x2039a426 (stale) Mirror 16: 0x2039a426 (stale) Mirror 17: 0x2039a426 (stale) Mirror 18: 0x2039a426 (stale) Mirror 19: 0x6e146dab (stale) CRC-32 checksum value for chunk [0, 0x301bb): Mirror 1: 0x4c34c91f Mirror 20: 0x4c34c91f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301bb: skipped 000671 1789479447.547422 mapwrite 0x00e895 thru 0x00e8dd (0x49 bytes) 000673 1789479447.750558 mapread 0x000c3c thru 0x001ef7 (0x12bc bytes) 000677 1789479447.788481 mapread 0x0069c0 thru 0x013b86 (0xd1c7 bytes) 000678 1789479447.809919 write 0x03c05a thru 0x03ffff (0x3fa6 bytes) 000679 1789479447.811027 trunc from 0x040000 to 0x0323a9 (0xdc56 bytes) 000680 1789479447.868294 mapread 0x0319f3 thru 0x0323a9 (0x9b7 bytes) 000683 1789479447.888737 mapwrite 0x037451 thru 0x03d9ce (0x657e bytes) 000684 1789479448.041536 write 0x013771 thru 0x01784f (0x40df bytes) 000687 1789479448.046586 read 0x00b3cb thru 0x016ad1 (0xb707 bytes) 000688 1789479448.050232 mapread 0x019885 thru 0x028996 (0xf112 bytes) 000689 1789479448.081656 read 0x024196 thru 0x02d590 (0x93fb bytes) 000690 1789479448.087531 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 16: 0xd2430ff1 (stale) Mirror 17: 0xd2430ff1 (stale) Mirror 18: 0xd2430ff1 (stale) Mirror 19: 0x5a7fa47d (stale) Mirror 20: 0x9a7a8220 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 16: 0xd2430ff1 (stale) Mirror 17: 0xd2430ff1 (stale) Mirror 18: 0xd2430ff1 (stale) Mirror 19: 0x5a7fa47d (stale) Mirror 20: 0x9a7a8220 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9cf: skipped 000691 1789479448.797464 mapread 0x03233f thru 0x03d9ce (0xb690 bytes) 000692 1789479448.914872 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 16: 0xd2430ff1 (stale) Mirror 17: 0xd2430ff1 (stale) Mirror 18: 0xd2430ff1 (stale) Mirror 19: 0x5a7fa47d (stale) Mirror 20: 0x9a7a8220 (stale) CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 1: 0x6d43cbf9 Mirror 21: 0x6d43cbf9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9cf: skipped 000693 1789479450.955538 mapread 0x005f24 thru 0x009f76 (0x4053 bytes) 000694 1789479451.018383 read 0x02cd9f thru 0x0393d6 (0xc638 bytes) 000695 1789479451.058485 mapwrite 0x012fc4 thru 0x01cfdd (0xa01a bytes) 000697 1789479451.329393 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 16: 0xd2430ff1 (stale) Mirror 17: 0xd2430ff1 (stale) Mirror 18: 0xd2430ff1 (stale) Mirror 19: 0x5a7fa47d (stale) Mirror 20: 0x9a7a8220 (stale) Mirror 21: 0x6d43cbf9 (stale) CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 1: 0x64050f17 Mirror 22: 0x64050f17 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9cf: skipped 000698 1789479452.855641 mapwrite 0x03551f thru 0x03782d (0x230f bytes) 000699 1789479453.177589 mapwrite 0x00787a thru 0x0154dc (0xdc63 bytes) 000700 1789479453.270540 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 16: 0xd2430ff1 (stale) Mirror 17: 0xd2430ff1 (stale) Mirror 18: 0xd2430ff1 (stale) Mirror 19: 0x5a7fa47d (stale) Mirror 20: 0x9a7a8220 (stale) Mirror 21: 0x6d43cbf9 (stale) Mirror 22: 0x64050f17 (stale) CRC-32 checksum value for chunk [0, 0x3d9cf): Mirror 1: 0xb1a3e212 Mirror 23: 0xb1a3e212 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9cf: skipped 000701 1789479454.914676 mapwrite 0x00183d thru 0x002fa5 (0x1769 bytes) 000702 1789479455.090434 trunc from 0x03d9cf to 0x0141fc (0x297d2 bytes) 000704 1789479455.111801 read 0x009a36 thru 0x0141fc (0xa7c7 bytes) 000705 1789479455.119271 read 0x001a3a thru 0x0074e0 (0x5aa7 bytes) 000707 1789479455.129596 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x141fd): Mirror 17: 0x1a91483f (stale) Mirror 18: 0x1a91483f (stale) Mirror 19: 0x14301d0e (stale) Mirror 20: 0x14301d0e (stale) Mirror 21: 0x60634fc1 (stale) Mirror 22: 0xcf6c4250 (stale) Mirror 23: 0x2928325a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x141fd): Mirror 17: 0x1a91483f (stale) Mirror 18: 0x1a91483f (stale) Mirror 19: 0x14301d0e (stale) Mirror 20: 0x14301d0e (stale) Mirror 21: 0x60634fc1 (stale) Mirror 22: 0xcf6c4250 (stale) Mirror 23: 0x2928325a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x141fd: skipped 000709 1789479455.529122 read 0x00a358 thru 0x010e62 (0x6b0b bytes) 000711 1789479455.540306 read 0x00d358 thru 0x00ef60 (0x1c09 bytes) 000712 1789479455.540997 mapread 0x0064ba thru 0x01072d (0xa274 bytes) 000715 1789479455.549153 mapwrite 0x03b8dd thru 0x03ffff (0x4723 bytes) 000716 1789479455.610226 trunc from 0x040000 to 0x02aec8 (0x15137 bytes) 000717 1789479455.633016 trunc from 0x02aec9 to 0x01db94 (0xd334 bytes) 000720 1789479455.651709 mapwrite 0x03f356 thru 0x03ffff (0xcaa bytes) 000722 1789479455.738427 write 0x016d4e thru 0x02386c (0xcb1f bytes) 000723 1789479455.756844 read 0x02f965 thru 0x03cee9 (0xd585 bytes) 000726 1789479455.770983 read 0x039877 thru 0x03ffff (0x6789 bytes) 000727 1789479455.771604 mapread 0x024a8e thru 0x02db24 (0x9097 bytes) 000728 1789479455.772541 mapread 0x038366 thru 0x03ffff (0x7c9a bytes) 000729 1789479455.779045 mapread 0x00d920 thru 0x013226 (0x5907 bytes) 000732 1789479455.793125 MIRROR_RESYNC 8 mirrors 000733 1789479457.604400 mapwrite 0x029789 thru 0x02c252 (0x2aca bytes) 000734 1789479457.745139 write 0x03204a thru 0x0388b2 (0x6869 bytes) 000736 1789479457.755333 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xda95af5f (stale) Mirror 19: 0xda95af5f (stale) Mirror 20: 0xda95af5f (stale) Mirror 21: 0xda95af5f (stale) Mirror 22: 0xda95af5f (stale) Mirror 23: 0xda95af5f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xda95af5f (stale) Mirror 19: 0xda95af5f (stale) Mirror 20: 0xda95af5f (stale) Mirror 21: 0xda95af5f (stale) Mirror 22: 0xda95af5f (stale) Mirror 23: 0xda95af5f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000737 1789479458.123514 mapread 0x01b0bb thru 0x01db77 (0x2abd bytes) 000738 1789479458.141319 mapwrite 0x026594 thru 0x0290ea (0x2b57 bytes) 000739 1789479458.182641 mapread 0x005a6e thru 0x0098ee (0x3e81 bytes) 000741 1789479458.191823 mapread 0x021c4a thru 0x024a7f (0x2e36 bytes) 000742 1789479458.192968 read 0x0122fd thru 0x01316e (0xe72 bytes) 000743 1789479458.198162 mapread 0x02f6f5 thru 0x03dbcf (0xe4db bytes) 000744 1789479458.213313 mapread 0x03ba26 thru 0x03e369 (0x2944 bytes) 000745 1789479458.214323 read 0x0069c1 thru 0x00ea78 (0x80b8 bytes) 000746 1789479458.220326 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xda95af5f (stale) Mirror 19: 0xda95af5f (stale) Mirror 20: 0xda95af5f (stale) Mirror 21: 0xda95af5f (stale) Mirror 22: 0xda95af5f (stale) Mirror 23: 0xda95af5f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xeffff56f Mirror 24: 0xeffff56f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000748 1789479458.914829 MIRROR_RESYNC 8 mirrors 000749 1789479459.971578 mapwrite 0x0178f7 thru 0x0238a6 (0xbfb0 bytes) 000750 1789479460.181603 trunc from 0x040000 to 0x031010 (0xefef bytes) 000752 1789479460.231332 write 0x02cdf3 thru 0x03c2ee (0xf4fc bytes) 000753 1789479460.238370 read 0x01bd5c thru 0x0270c8 (0xb36d bytes) 000754 1789479460.239348 mapread 0x02f6e3 thru 0x0354ed (0x5e0b bytes) 000756 1789479460.242048 mapwrite 0x00768c thru 0x0171e0 (0xfb55 bytes) 000757 1789479460.303542 mapread 0x016df2 thru 0x01a100 (0x330f bytes) 000758 1789479460.308251 mapwrite 0x028e5e thru 0x02e164 (0x5307 bytes) 000759 1789479460.352911 write 0x0373a9 thru 0x03df98 (0x6bf0 bytes) 000760 1789479460.354423 mapwrite 0x012f46 thru 0x01d971 (0xaa2c bytes) 000761 1789479460.441293 read 0x00efe1 thru 0x0123ac (0x33cc bytes) 000765 1789479460.452901 read 0x00fd1f thru 0x01810e (0x83f0 bytes) 000769 1789479460.455126 mapwrite 0x025a1f thru 0x0300fb (0xa6dd bytes) 000770 1789479460.612382 write 0x021d2f thru 0x025255 (0x3527 bytes) 000772 1789479460.614480 mapwrite 0x00a1bf thru 0x00b9da (0x181c bytes) 000773 1789479460.711806 mapwrite 0x01a37e thru 0x0234d9 (0x915c bytes) 000778 1789479460.783337 write 0x005a0a thru 0x00fa36 (0xa02d bytes) 000779 1789479460.792038 read 0x00afe6 thru 0x00b145 (0x160 bytes) 000781 1789479460.794021 mapwrite 0x013955 thru 0x01e474 (0xab20 bytes) 000782 1789479460.893242 mapread 0x038046 thru 0x03df98 (0x5f53 bytes) 000784 1789479460.901078 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3df99): Mirror 18: 0x429b8615 (stale) Mirror 19: 0x429b8615 (stale) Mirror 20: 0x429b8615 (stale) Mirror 21: 0x429b8615 (stale) Mirror 23: 0x429b8615 (stale) Mirror 24: 0x429b8615 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3df99): Mirror 18: 0x429b8615 (stale) Mirror 19: 0x429b8615 (stale) Mirror 20: 0x429b8615 (stale) Mirror 21: 0x429b8615 (stale) Mirror 23: 0x429b8615 (stale) Mirror 24: 0x429b8615 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3df99: skipped 000785 1789479461.393628 mapread 0x00ffb7 thru 0x018627 (0x8671 bytes) 000787 1789479461.411430 read 0x002aed thru 0x006246 (0x375a bytes) 000789 1789479461.419827 trunc from 0x03df99 to 0x01cb8d (0x2140b bytes) 000790 1789479461.434766 mapwrite 0x01171c thru 0x019499 (0x7d7e bytes) 000792 1789479461.497963 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1cb8e): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cb8e): Mirror 18: 0x6a68532b (stale) Mirror 19: 0x6a68532b (stale) Mirror 20: 0x6a68532b (stale) Mirror 21: 0x6a68532b (stale) Mirror 23: 0x6a68532b (stale) Mirror 24: 0x6a68532b (stale) CRC-32 checksum value for chunk [0, 0x1cb8e): Mirror 1: 0xf174160b Mirror 25: 0xf174160b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cb8e: skipped 000797 1789479462.260244 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1cb8e): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cb8e): Mirror 18: 0x6a68532b (stale) Mirror 19: 0x6a68532b (stale) Mirror 20: 0x6a68532b (stale) Mirror 21: 0x6a68532b (stale) Mirror 23: 0x6a68532b (stale) Mirror 24: 0x6a68532b (stale) CRC-32 checksum value for chunk [0, 0x1cb8e): Mirror 1: 0xf174160b Mirror 25: 0xf174160b Mirror 26: 0xf174160b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cb8e: skipped 000798 1789479463.407794 mapread 0x00c1b8 thru 0x00c70a (0x553 bytes) 000799 1789479463.413019 mapwrite 0x02b6ca thru 0x034204 (0x8b3b bytes) 000800 1789479463.598433 mapread 0x0097a3 thru 0x00b32d (0x1b8b bytes) 000804 1789479463.619949 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x34205): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34205): Mirror 18: 0x508e88c4 (stale) Mirror 19: 0x508e88c4 (stale) Mirror 20: 0x508e88c4 (stale) Mirror 21: 0x508e88c4 (stale) Mirror 23: 0x508e88c4 (stale) Mirror 24: 0x508e88c4 (stale) Mirror 25: 0x4684975c (stale) Mirror 26: 0x4684975c (stale) CRC-32 checksum value for chunk [0, 0x34205): Mirror 1: 0x48eb28a4 Mirror 27: 0x48eb28a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34205: skipped 000805 1789479464.429144 trunc from 0x034205 to 0x012890 (0x21974 bytes) 000807 1789479464.555245 write 0x02504d thru 0x0341d2 (0xf186 bytes) 000810 1789479464.559408 write 0x00145e thru 0x009833 (0x83d6 bytes) 000812 1789479464.571690 write 0x0166c9 thru 0x019a42 (0x337a bytes) 000813 1789479464.580519 mapwrite 0x02970d thru 0x03925a (0xfb4e bytes) 000815 1789479464.693399 read 0x00e24d thru 0x01ce7b (0xec2f bytes) 000816 1789479464.706061 mapwrite 0x00bdf6 thru 0x01a07f (0xe28a bytes) 000818 1789479464.752133 write 0x01fdc5 thru 0x027e79 (0x80b5 bytes) 000819 1789479464.761170 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3925b): Mirror 19: 0x63899769 (stale) Mirror 20: 0x63899769 (stale) Mirror 21: 0x63899769 (stale) Mirror 23: 0x63899769 (stale) Mirror 24: 0x63899769 (stale) Mirror 25: 0x3fd2a662 (stale) Mirror 26: 0x3fd2a662 (stale) Mirror 27: 0xfa96b647 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3925b): Mirror 19: 0x63899769 (stale) Mirror 20: 0x63899769 (stale) Mirror 21: 0x63899769 (stale) Mirror 23: 0x63899769 (stale) Mirror 24: 0x63899769 (stale) Mirror 25: 0x3fd2a662 (stale) Mirror 26: 0x3fd2a662 (stale) Mirror 27: 0xfa96b647 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3925b: skipped 000822 1789479465.201926 write 0x01a02c thru 0x0229fd (0x89d2 bytes) 000824 1789479465.212615 trunc from 0x03925b to 0x00eef5 (0x2a365 bytes) 000825 1789479465.264677 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xeef6): Mirror 19: 0x8f622a72 (stale) Mirror 20: 0x8f622a72 (stale) Mirror 21: 0x8f622a72 (stale) Mirror 24: 0x8f622a72 (stale) Mirror 25: 0x7e06159 (stale) Mirror 26: 0x7e06159 (stale) Mirror 27: 0x7e06159 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeef6): Mirror 19: 0x8f622a72 (stale) Mirror 20: 0x8f622a72 (stale) Mirror 21: 0x8f622a72 (stale) Mirror 24: 0x8f622a72 (stale) Mirror 25: 0x7e06159 (stale) Mirror 26: 0x7e06159 (stale) Mirror 27: 0x7e06159 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeef6: skipped 000826 1789479465.675319 write 0x033e42 thru 0x03b0cb (0x728a bytes) 000828 1789479465.678646 mapwrite 0x01f8dc thru 0x02c4a5 (0xcbca bytes) 000830 1789479465.742287 mapwrite 0x027089 thru 0x035289 (0xe201 bytes) 000831 1789479465.781121 read 0x024869 thru 0x0265f7 (0x1d8f bytes) 000833 1789479465.783320 write 0x03c919 thru 0x03ffff (0x36e7 bytes) 000834 1789479465.785003 mapwrite 0x038dcc thru 0x03ffff (0x7234 bytes) 000837 1789479465.830495 mapwrite 0x033682 thru 0x03ffff (0xc97e bytes) 000838 1789479465.886949 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xeffff56f (stale) Mirror 20: 0xeffff56f (stale) Mirror 21: 0xeffff56f (stale) Mirror 24: 0xeffff56f (stale) Mirror 25: 0xa14bcb66 (stale) Mirror 26: 0xa14bcb66 (stale) Mirror 27: 0x3a50b9a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x188b9798 Mirror 28: 0x188b9798 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000839 1789479466.796862 read 0x0248fb thru 0x029650 (0x4d56 bytes) 000840 1789479466.811389 mapread 0x03e5cc thru 0x03ffff (0x1a34 bytes) 000841 1789479466.838065 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xeffff56f (stale) Mirror 21: 0xeffff56f (stale) Mirror 24: 0xeffff56f (stale) Mirror 25: 0xa14bcb66 (stale) Mirror 26: 0xa14bcb66 (stale) Mirror 27: 0x3a50b9a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xeffff56f (stale) Mirror 21: 0xeffff56f (stale) Mirror 24: 0xeffff56f (stale) Mirror 25: 0xa14bcb66 (stale) Mirror 26: 0xa14bcb66 (stale) Mirror 27: 0x3a50b9a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x188b9798 Mirror 28: 0x188b9798 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000842 1789479467.669280 mapread 0x012f17 thru 0x01a17b (0x7265 bytes) 000843 1789479467.711864 trunc from 0x040000 to 0x03eb8d (0x1472 bytes) 000844 1789479467.831334 write 0x032d60 thru 0x035177 (0x2418 bytes) 000845 1789479467.842183 read 0x009489 thru 0x00f0f4 (0x5c6c bytes) 000846 1789479467.851083 mapwrite 0x01a2ae thru 0x01eda1 (0x4af4 bytes) 000847 1789479467.907836 read 0x01d43f thru 0x029d1e (0xc8e0 bytes) 000848 1789479467.923161 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3eb8e): Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb8e): Mirror 19: 0x529d9b3a (stale) Mirror 21: 0x529d9b3a (stale) Mirror 24: 0x529d9b3a (stale) Mirror 25: 0x11f152c3 (stale) Mirror 26: 0x11f152c3 (stale) Mirror 27: 0xe841029c (stale) Mirror 28: 0x92a41af9 (stale) CRC-32 checksum value for chunk [0, 0x3eb8e): Mirror 1: 0x4543308c Mirror 29: 0x4543308c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb8e: skipped 000849 1789479468.950416 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb8e): Mirror 19: 0x529d9b3a (stale) Mirror 24: 0x529d9b3a (stale) Mirror 25: 0x11f152c3 (stale) Mirror 26: 0x11f152c3 (stale) Mirror 27: 0xe841029c (stale) Mirror 28: 0x92a41af9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb8e): Mirror 19: 0x529d9b3a (stale) Mirror 24: 0x529d9b3a (stale) Mirror 25: 0x11f152c3 (stale) Mirror 26: 0x11f152c3 (stale) Mirror 27: 0xe841029c (stale) Mirror 28: 0x92a41af9 (stale) CRC-32 checksum value for chunk [0, 0x3eb8e): Mirror 1: 0x4543308c Mirror 29: 0x4543308c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb8e: skipped 000850 1789479469.936692 write 0x028ef4 thru 0x036a0b (0xdb18 bytes) 000857 1789479470.075951 write 0x01acb3 thru 0x01b2a9 (0x5f7 bytes) 000860 1789479470.095906 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3eb8e): Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb8e): Mirror 19: 0x529d9b3a (stale) Mirror 24: 0x529d9b3a (stale) Mirror 25: 0x11f152c3 (stale) Mirror 26: 0x11f152c3 (stale) Mirror 27: 0xe841029c (stale) Mirror 28: 0x92a41af9 (stale) Mirror 29: 0x4543308c (stale) CRC-32 checksum value for chunk [0, 0x3eb8e): Mirror 1: 0xff672257 Mirror 30: 0xff672257 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb8e: skipped 000861 1789479471.102093 read 0x015e37 thru 0x01b6ef (0x58b9 bytes) 000862 1789479471.117727 trunc from 0x03eb8e to 0x018b8b (0x26002 bytes) 000863 1789479471.214964 trunc from 0x018b8c to 0x0027ca (0x163c1 bytes) 000864 1789479471.235731 trunc from 0x0027cb to 0x00d3f4 (0xac2a bytes) 000865 1789479471.256432 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd3f5): Mirror 19: 0xe4e60d3f (stale) Mirror 24: 0xe4e60d3f (stale) Mirror 25: 0x67b2995f (stale) Mirror 27: 0x67b2995f (stale) Mirror 28: 0xded2f0f0 (stale) Mirror 29: 0xded2f0f0 (stale) Mirror 30: 0xded2f0f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd3f5): Mirror 19: 0xe4e60d3f (stale) Mirror 24: 0xe4e60d3f (stale) Mirror 25: 0x67b2995f (stale) Mirror 27: 0x67b2995f (stale) Mirror 28: 0xded2f0f0 (stale) Mirror 29: 0xded2f0f0 (stale) Mirror 30: 0xded2f0f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd3f5: skipped 000866 1789479471.679824 trunc from 0x00d3f5 to 0x0129de (0x55ea bytes) 000868 1789479471.694902 mapwrite 0x00994f thru 0x018851 (0xef03 bytes) 000870 1789479471.906996 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18852): Mirror 19: 0x88bf020e (stale) Mirror 24: 0x88bf020e (stale) Mirror 25: 0x72e7030 (stale) Mirror 27: 0x72e7030 (stale) Mirror 28: 0x4f463727 (stale) Mirror 29: 0x4f463727 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18852): Mirror 19: 0x88bf020e (stale) Mirror 24: 0x88bf020e (stale) Mirror 25: 0x72e7030 (stale) Mirror 27: 0x72e7030 (stale) Mirror 28: 0x4f463727 (stale) Mirror 29: 0x4f463727 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18852: skipped 000871 1789479472.296977 mapread 0x00572d thru 0x011a56 (0xc32a bytes) 000873 1789479472.325684 read 0x00e50f thru 0x017c0a (0x96fc bytes) 000874 1789479472.326317 read 0x001c40 thru 0x002d1d (0x10de bytes) 000876 1789479472.331600 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x18852): Mirror 19: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18852): Mirror 19: 0x88bf020e (stale) Mirror 24: 0x88bf020e (stale) Mirror 25: 0x72e7030 (stale) Mirror 27: 0x72e7030 (stale) Mirror 28: 0x4f463727 (stale) Mirror 29: 0x4f463727 (stale) CRC-32 checksum value for chunk [0, 0x18852): Mirror 1: 0x713706b4 Mirror 30: 0x713706b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18852: skipped 000879 1789479473.250269 write 0x01b9dc thru 0x02b487 (0xfaac bytes) 000882 1789479473.404040 MIRROR_RESYNC 8 mirrors 000884 1789479474.993591 mapwrite 0x01eecb thru 0x021013 (0x2149 bytes) 000885 1789479475.140927 trunc from 0x02b488 to 0x03d12f (0x11ca8 bytes) 000886 1789479475.157332 mapwrite 0x022632 thru 0x02d1a4 (0xab73 bytes) 000888 1789479475.200976 read 0x030f77 thru 0x037ed2 (0x6f5c bytes) 000890 1789479475.210194 mapwrite 0x0361a8 thru 0x03783c (0x1695 bytes) 000891 1789479475.253595 mapread 0x007867 thru 0x015540 (0xdcda bytes) 000892 1789479475.271207 mapwrite 0x00fb2d thru 0x01a522 (0xa9f6 bytes) 000893 1789479475.313970 mapread 0x00aabe thru 0x00c9d2 (0x1f15 bytes) 000894 1789479475.314850 mapwrite 0x01b416 thru 0x01c867 (0x1452 bytes) 000896 1789479475.355063 write 0x016800 thru 0x023241 (0xca42 bytes) 000897 1789479475.357180 mapwrite 0x020b47 thru 0x0288c9 (0x7d83 bytes) 000898 1789479475.429281 mapwrite 0x016f0f thru 0x01b2a1 (0x4393 bytes) 000899 1789479475.465945 trunc from 0x03d130 to 0x005ed2 (0x3725d bytes) 000900 1789479475.480659 mapread 0x002c22 thru 0x005ed2 (0x32b1 bytes) 000901 1789479475.483382 write 0x03f458 thru 0x03ffff (0xba8 bytes) 000902 1789479475.484988 mapread 0x020e4b thru 0x0238cb (0x2a81 bytes) 000903 1789479475.491565 write 0x024b73 thru 0x026792 (0x1c20 bytes) 000905 1789479475.492574 mapwrite 0x002b57 thru 0x005f8d (0x3437 bytes) 000909 1789479475.533237 mapread 0x0314b5 thru 0x03cbfe (0xb74a bytes) 000910 1789479475.550770 read 0x00f382 thru 0x019be9 (0xa868 bytes) 000911 1789479475.563412 mapread 0x006f45 thru 0x007b31 (0xbed bytes) 000914 1789479475.566125 write 0x02ee1f thru 0x0331af (0x4391 bytes) 000916 1789479475.568411 mapwrite 0x01256b thru 0x02222a (0xfcc0 bytes) 000917 1789479475.622728 trunc from 0x040000 to 0x0219d5 (0x1e62a bytes) 000918 1789479475.643076 write 0x038962 thru 0x03ffff (0x769e bytes) 000920 1789479475.651241 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xf75a53f1 (stale) Mirror 25: 0xf75a53f1 (stale) Mirror 27: 0xf75a53f1 (stale) Mirror 28: 0xf75a53f1 (stale) Mirror 29: 0xf75a53f1 (stale) Mirror 30: 0xf75a53f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xf75a53f1 (stale) Mirror 25: 0xf75a53f1 (stale) Mirror 27: 0xf75a53f1 (stale) Mirror 28: 0xf75a53f1 (stale) Mirror 29: 0xf75a53f1 (stale) Mirror 30: 0xf75a53f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000921 1789479476.091281 mapwrite 0x0197d0 thru 0x01d9d7 (0x4208 bytes) 000922 1789479476.157393 read 0x02a815 thru 0x0391c0 (0xe9ac bytes) 000925 1789479476.169187 mapwrite 0x01bdaf thru 0x01c3da (0x62c bytes) 000926 1789479476.212275 MIRROR_RESYNC 7 mirrors 000927 1789479477.654210 read 0x03700d thru 0x03c63a (0x562e bytes) 000929 1789479477.674771 mapwrite 0x0127da thru 0x020e7a (0xe6a1 bytes) 000931 1789479478.005025 mapread 0x00f6cf thru 0x01251a (0x2e4c bytes) 000933 1789479478.013946 read 0x02d994 thru 0x03b4bc (0xdb29 bytes) 000934 1789479478.030715 read 0x038091 thru 0x03ffff (0x7f6f bytes) 000937 1789479478.033950 mapread 0x0070a5 thru 0x00a672 (0x35ce bytes) 000938 1789479478.035621 mapwrite 0x03e75b thru 0x03ffff (0x18a5 bytes) 000939 1789479478.125294 trunc from 0x040000 to 0x00067b (0x3f984 bytes) 000942 1789479478.170776 mapwrite 0x016307 thru 0x02039f (0xa099 bytes) 000945 1789479478.302030 mapread 0x01ae06 thru 0x02039f (0x559a bytes) 000946 1789479478.302956 read 0x01d5c5 thru 0x01f699 (0x20d5 bytes) 000950 1789479478.308851 write 0x010711 thru 0x01b473 (0xad63 bytes) 000954 1789479478.323306 mapread 0x0094aa thru 0x009d00 (0x857 bytes) 000955 1789479478.325204 write 0x038153 thru 0x03a2b4 (0x2162 bytes) 000956 1789479478.327239 mapwrite 0x00aefa thru 0x00eb26 (0x3c2d bytes) 000957 1789479478.444187 trunc from 0x03a2b5 to 0x028e22 (0x11492 bytes) 000959 1789479478.478405 write 0x028d99 thru 0x02d25c (0x44c4 bytes) 000963 1789479478.537295 write 0x01c1dd thru 0x025814 (0x9638 bytes) 000965 1789479478.542077 mapwrite 0x01836f thru 0x02438d (0xc01f bytes) 000968 1789479478.620590 mapwrite 0x0275f3 thru 0x02e366 (0x6d74 bytes) 000972 1789479478.762697 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2e367): Mirror 19: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e367): Mirror 19: 0x76e5172c (stale) Mirror 25: 0x76e5172c (stale) Mirror 27: 0x76e5172c (stale) Mirror 28: 0x76e5172c (stale) Mirror 29: 0x76e5172c (stale) Mirror 30: 0x76e5172c (stale) CRC-32 checksum value for chunk [0, 0x2e367): Mirror 1: 0x37b32de0 Mirror 31: 0x37b32de0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e367: skipped 000973 1789479479.700471 trunc from 0x02e367 to 0x030dec (0x2a86 bytes) 000974 1789479479.821613 write 0x027285 thru 0x032467 (0xb1e3 bytes) 000975 1789479479.829639 write 0x004cb2 thru 0x00a752 (0x5aa1 bytes) 000977 1789479479.844785 mapwrite 0x01b995 thru 0x028658 (0xccc4 bytes) 000978 1789479480.053834 read 0x00883e thru 0x012e8f (0xa652 bytes) 000979 1789479480.067059 mapread 0x02964b thru 0x032467 (0x8e1d bytes) 000980 1789479480.069180 trunc from 0x032468 to 0x01ac55 (0x17812 bytes) 000981 1789479480.099357 trunc from 0x01ac56 to 0x00dc98 (0xcfbd bytes) 000983 1789479480.125859 read 0x003e28 thru 0x00c6cf (0x88a8 bytes) 000985 1789479480.136415 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdc99): Mirror 19: 0x61346055 (stale) Mirror 25: 0x61346055 (stale) Mirror 28: 0x61346055 (stale) Mirror 29: 0x61346055 (stale) Mirror 30: 0x61346055 (stale) Mirror 31: 0xf95f0288 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdc99): Mirror 19: 0x61346055 (stale) Mirror 25: 0x61346055 (stale) Mirror 28: 0x61346055 (stale) Mirror 29: 0x61346055 (stale) Mirror 30: 0x61346055 (stale) Mirror 31: 0xf95f0288 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdc99: skipped 000986 1789479480.559145 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1b5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdc99): Mirror 25: 0x61346055 (stale) Mirror 28: 0x61346055 (stale) Mirror 29: 0x61346055 (stale) Mirror 30: 0x61346055 (stale) Mirror 31: 0xf95f0288 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdc99): Mirror 25: 0x61346055 (stale) Mirror 28: 0x61346055 (stale) Mirror 29: 0x61346055 (stale) Mirror 30: 0x61346055 (stale) Mirror 31: 0xf95f0288 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdc99: skipped 000989 1789479481.122602 mapwrite 0x004843 thru 0x005a66 (0x1224 bytes) 000991 1789479481.178590 trunc from 0x00dc99 to 0x037626 (0x2998e bytes) 000996 1789479481.196320 trunc from 0x037627 to 0x02cc5d (0xa9c9 bytes) 000998 1789479481.209845 mapwrite 0x012e6f thru 0x0148d2 (0x1a64 bytes) 000999 1789479481.261948 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2cc5e): Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cc5e): Mirror 25: 0x23bae4cd (stale) Mirror 28: 0x23bae4cd (stale) Mirror 29: 0x23bae4cd (stale) Mirror 30: 0x23bae4cd (stale) Mirror 31: 0x55af5e58 (stale) CRC-32 checksum value for chunk [0, 0x2cc5e): Mirror 1: 0x4abc8cb3 Mirror 32: 0x4abc8cb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cc5e: skipped 001000 1789479482.019865 read 0x01bae6 thru 0x028741 (0xcc5c bytes) All operations completed A-OK! PASS 70a (115s) == sanity-flr test 71: check mirror extend parent fid ==== 09:38:08 (1789479488) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 09:38:11 (1789479491) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.19628 s, 10.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186081 s, 5.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 09:38:20 (1789479500) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4846112 .. Extending file size to 4586688 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8608064 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5255136 .. Extending file size to 7539456 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8372864 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9010528 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 57508 57510 57511 57512 57515 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) 192.168.202.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:39:40 (1789479580) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7956608 .. Extending file size to 4711360 .. failed Extending file size to 6178720 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7603936 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8987520 .. Extending file size to 8431072 .. Extending file size to 9098176 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9105760 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 59056 59057 59058 59062 59063 failed 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) 192.168.202.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:41:03 (1789479663) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0906488 s, 23.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.284585 s, 7.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa32c472d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:41:24 (1789479684) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0832734 s, 25.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 09:41:31 (1789479691) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x20a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:41:38 (1789479698) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199903 s, 10.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:41:47 (1789479707) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.242108 s, 8.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:41:55 (1789479715) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.312107 s, 6.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:42:29 (1789479749) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg232-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109565 s, 9.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:43:26 (1789479806) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg232-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16952 s, 12.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:44:25 (1789479865) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg232-server mds-ost sync done. Creating new pool test_204f oleg232-server: Pool lustre.test_204f created Adding targets to pool oleg232-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg232-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.272456 s, 7.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg232-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg232-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg232-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:45:45 (1789479945) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:45:54 (1789479954) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:46:03 (1789479963) oleg232-server: Pool lustre.test_206 created Adding targets to pool oleg232-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg232-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg232-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg232-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg232-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:46:28 (1789479988) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460158 s, 22.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103663 s, 10.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x75ef1713 Mirror 3: 0x75ef1713 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:46:42 (1789480002) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:46:46 (1789480006) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:46:51 (1789480011) Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.10551 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2200000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a00000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a00000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a00000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x13b0000 Read this (4 bytes): BBBB 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:47:26 (1789480046) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0321137 s, 255 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:47:34 (1789480054) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595937 s, 17.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x22f lmm_fid: [0x200000404:0x22f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x22f lmm_fid: [0x200000404:0x22f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834479 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-15 09:47:36.000000000 -0400 Modify: 2026-09-15 09:47:36.000000000 -0400 Change: 2026-09-15 09:47:38.000000000 -0400 Birth: 2026-09-15 09:47:36.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:47:44 (1789480064) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:47:47 (1789480067) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.927367 s, 44.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0332871 s, 123 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 15 09:47 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x231:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 15 09:47 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:47:58 (1789480078) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:48:06 (1789480086) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1914 sec =========== 09:48:16 (1789480096) === sanity-flr: start cleanup 09:48:18 (1789480098) === === sanity-flr: finish cleanup 09:48:26 (1789480106) ===