-----============= acceptance-small: sanity-flr ============----- Fri Aug 21 20:55:59 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg204-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 20:56:20 (1787360180) === oleg204-client.virtnet: executing check_config_client /mnt/lustre oleg204-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg204-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c2f52980800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c2f52980800.idle_timeout=debug disable quota as required oleg204-server: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 20:56:57 (1787360217) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7632] == sanity-flr test 0a: lfs mirror create with -N option == 20:57:00 (1787360220) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:57:19 (1787360239) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:57:23 (1787360243) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:57:27 (1787360247) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:57:56 (1787360276) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:58:00 (1787360280) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 20:58:04 (1787360284) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c2f52980800.stats=clear osc.lustre-OST0001-osc-ffff9c2f52980800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:58:12 (1787360292) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:58:24 (1787360304) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:58:32 (1787360312) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:58:36 (1787360316) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:58:44 (1787360324) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 20:58:55 (1787360335) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:59:02 (1787360342) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:59:05 (1787360345) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:59:13 (1787360353) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:59:24 (1787360364) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:59:31 (1787360371) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216986 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113488 s, 9.2 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:59:41 (1787360381) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.589694 s, 28.5 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:59:51 (1787360391) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:00:03 (1787360403) Stopping /mnt/lustre-ost1 (opts:) on oleg204-server oleg204-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2f52980800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2f52980800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg204-server: oleg204-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg204-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg204-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg204-server oleg204-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2f52980800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2f52980800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg204-server: oleg204-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg204-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (136s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:02:18 (1787360538) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.536753 s, 21.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg204-server oleg204-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2f52980800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2f52980800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg204-server: oleg204-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg204-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:03:02 (1787360582) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg204-server oleg204-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2f52980800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2f52980800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg204-server: oleg204-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg204-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg204-server oleg204-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2f52980800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2f52980800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg204-server: oleg204-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg204-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:04:22 (1787360662) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.84451 s, 10.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 21 21:04 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.112s user 0m0.000s sys 0m0.170s Stopping /mnt/lustre-ost1 (opts:) on oleg204-server oleg204-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2f52980800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2f52980800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.212s user 0m0.001s sys 0m0.177s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg204-server: oleg204-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg204-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg204-server oleg204-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2f52980800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2f52980800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.023s user 0m0.002s sys 0m0.160s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg204-server: oleg204-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg204-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg204-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c2f52980800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2f52980800.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:05:58 (1787360758) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:06:01 (1787360761) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:06:05 (1787360765) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 21:06:09 (1787360769) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 21:06:17 (1787360777) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.652293 s, 19.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:06:28 (1787360788) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521546 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028851 s, 36.3 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:06:35 (1787360795) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:06:43 (1787360803) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000949448 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000819163 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00159561 s, 642 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00622587 s, 164 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 21:06:52 (1787360812) create test files with size 15733159 .. f71631cfe68d490814f170761f50182e - 27eb322458737c73f0ca5d3ecc463b00 - 7e188f461c753dcd5e12a65ed266ae99 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg204-client.virtnet /mnt/lustre (opts:) Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 21:07:15 (1787360835) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:07:32 (1787360852) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:07:40 (1787360860) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142979 s, 7.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb5:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482027 s, 21.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 21:07:50 (1787360870) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25159 s, 8.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.423586 s, 9.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18836 s, 11.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 21:08:08 (1787360888) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 21:08:11 (1787360891) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:08:14 (1787360894) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388756 s, 27.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 21:08:22 (1787360902) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:08:26 (1787360906) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xbe:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xef:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xef:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:08:38 (1787360918) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.704294 s, 14.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg204-client.virtnet /mnt/lustre (opts:) Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:08:51 (1787360931) llite.lustre-ffff9c2f582f4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.308045 s, 34.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg204-client.virtnet /mnt/lustre (opts:) Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9c2f42bc8800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:09:06 (1787360946) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:09:49 (1787360989) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:10:33 (1787361033) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 21:10:38 (1787361038) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.388372 s, 8.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.303895 s, 10.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9255d40e (nosync,stale) Mirror 2: 0x9255d40e (nosync,stale) ** verify mirror 2 doesn't change original checksum: d6aeae01111d0110073105db4c9398bf - mirror 1 checksum: 6904670ec2ef9d5f4517750d2a6e9a88 - mirror 2 checksum: d6aeae01111d0110073105db4c9398bf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-21T21:10:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-21T21:10:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d6aeae01111d0110073105db4c9398bf - mirror 1 checksum: 6904670ec2ef9d5f4517750d2a6e9a88 - mirror 2 checksum: 6904670ec2ef9d5f4517750d2a6e9a88 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-21T21:10:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-21T21:10:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5:0x0] } PASS 48 (14s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:10:54 (1787361054) Failing mds1 on oleg204-server Stopping /mnt/lustre-mds1 (opts:) on oleg204-server 21:11:01 (1787361061) shut down facet: mds1 facet_host: oleg204-server facet_failover_host: oleg204-server Failover mds1 to oleg204-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Started lustre-MDT0000 21:11:25 (1787361085) targets are mounted 21:11:25 (1787361085) facet_failover done oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024373 s, 43.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:11:43 (1787361103) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:11:46 (1787361106) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:11:50 (1787361110) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00536947 s, 3.1 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:11:58 (1787361118) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:12:01 (1787361121) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00701997 s, 583 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0782302 s, 52.4 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:12:09 (1787361129) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:12:31 (1787361151) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:12:58 (1787361178) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:13:23 (1787361203) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:13:47 (1787361227) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 21:13:55 (1787361235) mirror create pid 54341 mirror split pid 54342 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54341 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54342 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 21:15:02 (1787361302) Seed set to 1304 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1787361304.759716 write 0x02481f thru 0x026b2b (0x230d bytes) truncating to largest ever: 0x54c2 000004 1787361304.771957 trunc from 0x026b2c to 0x0054c1 (0x2166a bytes) 000005 1787361304.807778 mapread 0x00228b thru 0x003254 (0xfca bytes) 000007 1787361304.821881 mapwrite 0x03d88a thru 0x03ffff (0x2776 bytes) 000010 1787361304.960885 mapread 0x020402 thru 0x028dc1 (0x89c0 bytes) truncating to largest ever: 0x3f618 000011 1787361304.974581 trunc from 0x040000 to 0x03f617 (0x9e8 bytes) 000013 1787361304.993700 trunc from 0x03f618 to 0x037e24 (0x77f3 bytes) 000014 1787361305.024035 mapread 0x01fa47 thru 0x0201ef (0x7a9 bytes) 000015 1787361305.026961 read 0x03438a thru 0x037e24 (0x3a9b bytes) 000017 1787361305.039534 read 0x00bc79 thru 0x01bc27 (0xffaf bytes) truncating to largest ever: 0x3fdb8 000018 1787361305.057303 trunc from 0x037e25 to 0x03fdb7 (0x7f93 bytes) 000019 1787361305.085273 trunc from 0x03fdb8 to 0x02894e (0x17469 bytes) 000020 1787361305.130963 mapread 0x00c9b6 thru 0x01b203 (0xe84e bytes) 000021 1787361305.134904 write 0x00535c thru 0x00f3d9 (0xa07e bytes) 000022 1787361305.138345 mapwrite 0x00712e thru 0x0121e4 (0xb0b7 bytes) 000023 1787361305.292821 write 0x00b238 thru 0x00c52f (0x12f8 bytes) 000025 1787361305.299564 MIRROR_EXTEND to 2 mirrors 000027 1787361306.589085 MIRROR_RESYNC 2 mirrors 000028 1787361307.013349 mapread 0x012827 thru 0x01a974 (0x814e bytes) 000030 1787361307.059892 mapread 0x027fea thru 0x02894e (0x965 bytes) 000034 1787361307.108641 mapread 0x00a4fc thru 0x00eee7 (0x49ec bytes) 000035 1787361307.136372 MIRROR_EXTEND to 3 mirrors 000036 1787361308.037878 mapread 0x021f0e thru 0x02894e (0x6a41 bytes) 000037 1787361308.066256 mapwrite 0x037c29 thru 0x038d0b (0x10e3 bytes) 000038 1787361308.197661 write 0x01c6a9 thru 0x02309a (0x69f2 bytes) 000040 1787361308.209153 write 0x00e58f thru 0x01867e (0xa0f0 bytes) 000041 1787361308.220909 mapread 0x025156 thru 0x027c3c (0x2ae7 bytes) 000042 1787361308.238663 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38d0c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d0c): Mirror 2: 0x1fcf7f44 (stale) Mirror 3: 0x1fcf7f44 (stale) CRC-32 checksum value for chunk [0, 0x38d0c): Mirror 1: 0xe742c673 Mirror 4: 0xe742c673 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d0c: skipped 000043 1787361309.440894 mapwrite 0x003fc2 thru 0x010f59 (0xcf98 bytes) 000045 1787361309.607073 read 0x00dcf5 thru 0x01b558 (0xd864 bytes) 000047 1787361309.620147 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38d0c): Mirror 3: 0x1fcf7f44 (stale) Mirror 4: 0xe742c673 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d0c): Mirror 3: 0x1fcf7f44 (stale) Mirror 4: 0xe742c673 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d0c: skipped 000048 1787361310.328105 write 0x028905 thru 0x02ec3f (0x633b bytes) 000050 1787361310.354545 read 0x0196c7 thru 0x01d2c2 (0x3bfc bytes) 000051 1787361310.369967 mapwrite 0x0087cd thru 0x00eb98 (0x63cc bytes) 000052 1787361310.546816 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38d0c): Mirror 3: 0x1fcf7f44 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d0c): Mirror 3: 0x1fcf7f44 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d0c: skipped 000054 1787361311.003271 mapwrite 0x026954 thru 0x026c34 (0x2e1 bytes) 000056 1787361311.055466 trunc from 0x038d0c to 0x021a35 (0x172d6 bytes) 000059 1787361311.099759 trunc from 0x021a36 to 0x002308 (0x1f72d bytes) 000060 1787361311.126358 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2309): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2309): Mirror 3: 0xc59008e4 (stale) CRC-32 checksum value for chunk [0, 0x2309): Mirror 1: 0xc59008e4 Mirror 4: 0xc59008e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2309: skipped 000063 1787361312.332586 MIRROR_RESYNC 3 mirrors 000065 1787361313.319898 mapread 0x000f84 thru 0x002308 (0x1385 bytes) 000066 1787361313.359125 mapwrite 0x018873 thru 0x022133 (0x98c1 bytes) 000067 1787361313.634043 write 0x0257a2 thru 0x0339bc (0xe21b bytes) 000069 1787361313.637174 trunc from 0x0339bd to 0x02ff0d (0x3aaf bytes) 000071 1787361313.672864 read 0x01860b thru 0x019855 (0x124b bytes) 000072 1787361313.674053 write 0x034bf5 thru 0x03f0dd (0xa4e9 bytes) 000074 1787361313.676389 write 0x025ec3 thru 0x0263dc (0x51a bytes) 000075 1787361313.677877 trunc from 0x03f0de to 0x039756 (0x5987 bytes) 000076 1787361313.696055 write 0x0194b5 thru 0x025c68 (0xc7b4 bytes) 000077 1787361313.698980 mapread 0x01d4ba thru 0x022a3d (0x5584 bytes) 000080 1787361313.709323 MIRROR_RESYNC 3 mirrors 000081 1787361315.045067 mapwrite 0x020293 thru 0x028afb (0x8869 bytes) 000082 1787361315.185571 write 0x026958 thru 0x032512 (0xbbbb bytes) 000083 1787361315.191420 write 0x00e95b thru 0x011d99 (0x343f bytes) 000084 1787361315.200852 trunc from 0x039757 to 0x03744d (0x2309 bytes) 000086 1787361315.228103 mapwrite 0x02e7ee thru 0x034766 (0x5f79 bytes) 000087 1787361315.296319 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3744e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3744e): Mirror 3: 0xddb9ad58 (stale) Mirror 4: 0xddb9ad58 (stale) CRC-32 checksum value for chunk [0, 0x3744e): Mirror 1: 0xb72556ae Mirror 5: 0xb72556ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3744e: skipped 000088 1787361316.220046 mapwrite 0x001473 thru 0x00aefc (0x9a8a bytes) 000090 1787361316.530323 mapwrite 0x00bcf2 thru 0x012d64 (0x7073 bytes) 000092 1787361316.661719 write 0x01e60d thru 0x024203 (0x5bf7 bytes) 000096 1787361316.678285 trunc from 0x03744e to 0x00e188 (0x292c5 bytes) 000097 1787361316.758811 MIRROR_RESYNC 4 mirrors 000098 1787361317.975302 mapread 0x00981a thru 0x00e188 (0x496f bytes) 000100 1787361318.042131 mapwrite 0x00d922 thru 0x00f15e (0x183d bytes) 000102 1787361318.417607 write 0x01be84 thru 0x023c29 (0x7da6 bytes) 000107 1787361318.448395 write 0x02e15a thru 0x0368fe (0x87a5 bytes) 000108 1787361318.459354 read 0x01ac4b thru 0x01dbf6 (0x2fac bytes) 000109 1787361318.469968 mapread 0x0167a1 thru 0x01845b (0x1cbb bytes) 000110 1787361318.494416 read 0x0258f5 thru 0x0347bc (0xeec8 bytes) 000117 1787361318.535293 read 0x0147a2 thru 0x0188fe (0x415d bytes) 000119 1787361318.538052 read 0x0139ef thru 0x0159e9 (0x1ffb bytes) 000121 1787361318.545429 read 0x006470 thru 0x010743 (0xa2d4 bytes) 000122 1787361318.550537 mapwrite 0x0130c8 thru 0x018785 (0x56be bytes) 000123 1787361318.669364 mapread 0x0234fe thru 0x02fbac (0xc6af bytes) 000125 1787361318.674585 mapread 0x01860f thru 0x025d62 (0xd754 bytes) 000126 1787361318.675589 trunc from 0x0368ff to 0x034edb (0x1a23 bytes) 000128 1787361318.697223 mapread 0x02b62f thru 0x034edb (0x98ad bytes) 000129 1787361318.698117 write 0x00b697 thru 0x00dc65 (0x25cf bytes) 000130 1787361318.703847 write 0x03bed1 thru 0x03ca9d (0xbcd bytes) 000132 1787361318.711859 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ca9e): Mirror 3: 0xc6c83ff5 (stale) Mirror 5: 0xc6c83ff5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca9e): Mirror 3: 0xc6c83ff5 (stale) Mirror 5: 0xc6c83ff5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca9e: skipped 000133 1787361319.154758 trunc from 0x03ca9e to 0x009a96 (0x33007 bytes) 000134 1787361319.173392 trunc from 0x009a97 to 0x02d2f3 (0x2385d bytes) 000136 1787361319.210124 mapwrite 0x000a4b thru 0x0020ff (0x16b5 bytes) 000137 1787361319.275704 mapwrite 0x034c6f thru 0x03ffff (0xb391 bytes) 000140 1787361319.494134 write 0x0070f3 thru 0x00fd40 (0x8c4e bytes) 000143 1787361319.514482 write 0x02fa0e thru 0x034cb2 (0x52a5 bytes) 000144 1787361319.516338 mapread 0x01a502 thru 0x01cf01 (0x2a00 bytes) 000145 1787361319.527031 mapwrite 0x0185d1 thru 0x021dc5 (0x97f5 bytes) 000146 1787361319.698852 mapwrite 0x03b312 thru 0x03ffff (0x4cee bytes) 000147 1787361319.787926 trunc from 0x040000 to 0x01d84f (0x227b0 bytes) 000148 1787361319.821089 write 0x02eb5d thru 0x031345 (0x27e9 bytes) 000149 1787361319.834018 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31346): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31346): Mirror 3: 0x59c2712c (stale) Mirror 5: 0x59c2712c (stale) CRC-32 checksum value for chunk [0, 0x31346): Mirror 1: 0x2c6029aa Mirror 6: 0x2c6029aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31346: skipped 000150 1787361321.346303 mapread 0x025b07 thru 0x026bbf (0x10b9 bytes) 000151 1787361321.393791 mapread 0x014f1f thru 0x020198 (0xb27a bytes) 000152 1787361321.477492 write 0x010e2e thru 0x01f417 (0xe5ea bytes) 000153 1787361321.623075 trunc from 0x031346 to 0x02a964 (0x69e1 bytes) 000154 1787361321.668682 trunc from 0x02a965 to 0x02be34 (0x14d0 bytes) 000155 1787361321.702672 mapwrite 0x03162d thru 0x0329d8 (0x13ac bytes) 000157 1787361321.891296 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x329d9): Mirror 3: 0x35bfa750 (stale) Mirror 5: 0x35bfa750 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x329d9): Mirror 3: 0x35bfa750 (stale) Mirror 5: 0x35bfa750 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x329d9: skipped 000158 1787361322.398676 write 0x026fb8 thru 0x02ccde (0x5d27 bytes) 000159 1787361322.413133 mapwrite 0x035b36 thru 0x03ea7b (0x8f46 bytes) 000160 1787361322.511886 read 0x00b318 thru 0x01958a (0xe273 bytes) 000161 1787361322.534612 MIRROR_RESYNC 3 mirrors 000163 1787361324.540924 MIRROR_EXTEND to 4 mirrors 000165 1787361326.001745 read 0x01b9c4 thru 0x0211fe (0x583b bytes) 000166 1787361326.015153 mapwrite 0x0256b3 thru 0x02ec94 (0x95e2 bytes) 000167 1787361326.332971 trunc from 0x03ea7c to 0x00095b (0x3e120 bytes) 000168 1787361326.363954 read 0x0004e4 thru 0x00095b (0x478 bytes) 000173 1787361326.385747 write 0x02c944 thru 0x02fd50 (0x340d bytes) 000174 1787361326.389680 write 0x0326b0 thru 0x03ffff (0xd950 bytes) 000175 1787361326.391716 read 0x033872 thru 0x035c6c (0x23fb bytes) 000176 1787361326.397292 MIRROR_RESYNC 4 mirrors 000177 1787361327.796934 trunc from 0x040000 to 0x0245fb (0x1ba04 bytes) 000179 1787361327.857687 write 0x02bb9d thru 0x02ffd3 (0x4437 bytes) 000182 1787361327.860306 mapread 0x0161b7 thru 0x022d1a (0xcb64 bytes) 000183 1787361327.882127 trunc from 0x02ffd4 to 0x01d68a (0x12949 bytes) 000184 1787361327.916771 mapread 0x00edbd thru 0x013b2d (0x4d71 bytes) 000185 1787361327.945651 write 0x037ea8 thru 0x03ffff (0x8158 bytes) 000189 1787361327.952843 mapwrite 0x0118f6 thru 0x015c22 (0x432d bytes) 000191 1787361328.024134 read 0x02dba6 thru 0x038616 (0xaa71 bytes) 000192 1787361328.039004 MIRROR_RESYNC 4 mirrors 000193 1787361329.871340 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000194 1787361330.405018 write 0x016701 thru 0x01b3e3 (0x4ce3 bytes) 000195 1787361330.560003 trunc from 0x040000 to 0x03f4c2 (0xb3d bytes) 000197 1787361330.588124 mapwrite 0x0084fe thru 0x0098dc (0x13df bytes) 000198 1787361330.674457 trunc from 0x03f4c3 to 0x0182d9 (0x271e9 bytes) 000199 1787361330.691650 mapwrite 0x03279e thru 0x03ab4e (0x83b1 bytes) 000203 1787361330.769563 write 0x0347da thru 0x03ffff (0xb826 bytes) 000205 1787361330.773492 trunc from 0x040000 to 0x01c1ba (0x23e45 bytes) 000206 1787361330.806291 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c1bb): Mirror 6: 0xa4982323 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c1bb): Mirror 6: 0xa4982323 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c1bb: skipped 000208 1787361331.100994 read 0x0015de thru 0x00949e (0x7ec1 bytes) 000212 1787361331.111915 mapread 0x0073ab thru 0x00f981 (0x85d7 bytes) 000213 1787361331.131624 mapwrite 0x014cba thru 0x01c342 (0x7689 bytes) 000214 1787361331.211244 mapread 0x002af9 thru 0x011a5d (0xef65 bytes) 000216 1787361331.212762 mapread 0x010f17 thru 0x01c342 (0xb42c bytes) 000220 1787361331.220858 mapread 0x00f284 thru 0x0117e5 (0x2562 bytes) 000222 1787361331.223818 read 0x0002c3 thru 0x00f018 (0xed56 bytes) 000223 1787361331.230501 trunc from 0x01c343 to 0x01804b (0x42f7 bytes) 000224 1787361331.254678 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1804c): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1804c): Mirror 6: 0xb91d1a25 (stale) CRC-32 checksum value for chunk [0, 0x1804c): Mirror 1: 0x3b19654f Mirror 7: 0x3b19654f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1804c: skipped 000225 1787361332.464502 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1804c): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1804c): Mirror 6: 0xb91d1a25 (stale) CRC-32 checksum value for chunk [0, 0x1804c): Mirror 1: 0x3b19654f Mirror 7: 0x3b19654f Mirror 8: 0x3b19654f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1804c: skipped 000226 1787361333.750692 trunc from 0x01804c to 0x028302 (0x102b7 bytes) 000227 1787361333.862718 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28303): Mirror 7: 0x26128267 (stale) Mirror 8: 0x26128267 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28303): Mirror 7: 0x26128267 (stale) Mirror 8: 0x26128267 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28303: skipped 000228 1787361334.224873 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28303): Mirror 8: 0x26128267 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28303): Mirror 8: 0x26128267 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28303: skipped 000229 1787361334.516952 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000230 1787361334.630469 write 0x037efa thru 0x03ffff (0x8106 bytes) 000232 1787361334.632235 mapread 0x030b7a thru 0x03c2fc (0xb783 bytes) 000233 1787361334.651070 trunc from 0x040000 to 0x0200e8 (0x1ff17 bytes) 000234 1787361334.666955 write 0x02f6b0 thru 0x030dfb (0x174c bytes) 000235 1787361334.668210 read 0x01d66f thru 0x028234 (0xabc6 bytes) 000236 1787361334.678540 trunc from 0x030dfc to 0x004f4c (0x2beaf bytes) 000237 1787361334.701101 mapread 0x0016a1 thru 0x004f4c (0x38ac bytes) 000239 1787361334.709055 mapread 0x002bbf thru 0x004f4c (0x238e bytes) 000242 1787361334.714425 trunc from 0x004f4d to 0x01acbc (0x15d70 bytes) 000243 1787361334.733818 write 0x00b7c6 thru 0x011dfe (0x6639 bytes) 000244 1787361334.746691 read 0x01154c thru 0x01acbc (0x9771 bytes) 000247 1787361334.758545 write 0x020fa9 thru 0x02dd0b (0xcd63 bytes) 000248 1787361334.760516 trunc from 0x02dd0c to 0x027edc (0x5e2f bytes) 000251 1787361334.778105 read 0x01b1c2 thru 0x027edc (0xcd1b bytes) 000254 1787361334.790963 trunc from 0x027edd to 0x0126ce (0x1580e bytes) 000255 1787361334.806682 read 0x00c506 thru 0x010574 (0x406f bytes) 000257 1787361334.813855 MIRROR_EXTEND to 2 mirrors 000260 1787361335.668099 read 0x011eb4 thru 0x0126ce (0x81b bytes) 000263 1787361335.739821 read 0x00c9b9 thru 0x01082a (0x3e72 bytes) 000264 1787361335.777381 mapread 0x008cb6 thru 0x0126ce (0x9a19 bytes) 000266 1787361335.830282 MIRROR_RESYNC 2 mirrors 000267 1787361336.265372 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000269 1787361336.496123 mapread 0x00c73c thru 0x00ffac (0x3871 bytes) 000272 1787361336.517990 mapwrite 0x031351 thru 0x031f55 (0xc05 bytes) 000273 1787361336.604962 read 0x01b9a3 thru 0x01da69 (0x20c7 bytes) 000276 1787361336.616025 mapread 0x022e38 thru 0x031ea3 (0xf06c bytes) 000277 1787361336.628608 write 0x02e614 thru 0x03206f (0x3a5c bytes) 000278 1787361336.629957 read 0x00269f thru 0x0077d5 (0x5137 bytes) 000279 1787361336.640607 write 0x031d4b thru 0x03c976 (0xac2c bytes) 000281 1787361336.643948 trunc from 0x03c977 to 0x01bcd7 (0x20c9f bytes) 000284 1787361336.669909 mapwrite 0x03c6c8 thru 0x03e99c (0x22d5 bytes) 000289 1787361336.761342 read 0x020f9d thru 0x02f342 (0xe3a6 bytes) 000291 1787361336.774809 read 0x00a04d thru 0x013c9d (0x9c51 bytes) 000297 1787361336.783153 read 0x00ce35 thru 0x016b31 (0x9cfd bytes) 000300 1787361336.786922 mapread 0x03102b thru 0x038ffd (0x7fd3 bytes) 000301 1787361336.791719 read 0x024a18 thru 0x03489b (0xfe84 bytes) 000302 1787361336.792758 write 0x03cbe7 thru 0x03ffff (0x3419 bytes) 000303 1787361336.794113 mapread 0x0041a4 thru 0x009984 (0x57e1 bytes) 000306 1787361336.799035 mapwrite 0x0130c9 thru 0x01c678 (0x95b0 bytes) 000307 1787361336.938048 MIRROR_EXTEND to 2 mirrors 000308 1787361338.020881 write 0x000eb1 thru 0x00cf01 (0xc051 bytes) 000309 1787361338.099600 read 0x015016 thru 0x024499 (0xf484 bytes) 000310 1787361338.115918 MIRROR_RESYNC 2 mirrors 000312 1787361338.961051 read 0x00ddd4 thru 0x00def3 (0x120 bytes) 000313 1787361338.965837 trunc from 0x040000 to 0x03abc2 (0x543d bytes) 000314 1787361339.079765 mapread 0x003595 thru 0x004b99 (0x1605 bytes) 000316 1787361339.089623 write 0x016326 thru 0x01a99b (0x4676 bytes) 000321 1787361339.110663 MIRROR_RESYNC 2 mirrors 000322 1787361339.744864 read 0x014907 thru 0x0175de (0x2cd8 bytes) 000323 1787361339.750565 trunc from 0x03abc3 to 0x035c8c (0x4f36 bytes) 000324 1787361339.833585 write 0x00df5e thru 0x015992 (0x7a35 bytes) 000325 1787361339.841288 mapwrite 0x03ef1e thru 0x03ffff (0x10e2 bytes) 000326 1787361339.915120 mapwrite 0x02cd6e thru 0x02e244 (0x14d7 bytes) 000328 1787361339.964497 mapread 0x03be2f thru 0x03ffff (0x41d1 bytes) 000330 1787361339.977984 MIRROR_RESYNC 2 mirrors 000334 1787361340.874468 mapwrite 0x02738f thru 0x02d014 (0x5c86 bytes) 000339 1787361341.067252 read 0x035b27 thru 0x03b84c (0x5d26 bytes) 000341 1787361341.090637 MIRROR_RESYNC 2 mirrors 000342 1787361341.962747 read 0x01253f thru 0x015674 (0x3136 bytes) 000343 1787361341.970237 mapwrite 0x0156b7 thru 0x01bf51 (0x689b bytes) 000344 1787361342.126837 mapwrite 0x007a40 thru 0x009b50 (0x2111 bytes) 000345 1787361342.187061 mapwrite 0x01acb4 thru 0x01ff70 (0x52bd bytes) 000347 1787361342.232999 mapwrite 0x007518 thru 0x00dfa2 (0x6a8b bytes) 000348 1787361342.281353 trunc from 0x040000 to 0x02d253 (0x12dac bytes) 000349 1787361342.306964 read 0x01b53e thru 0x02b3a5 (0xfe68 bytes) 000353 1787361342.319389 mapread 0x0242ce thru 0x02592b (0x165e bytes) 000357 1787361342.322648 mapread 0x0131d6 thru 0x01b9ff (0x882a bytes) 000358 1787361342.323742 mapread 0x00d7d9 thru 0x0125fa (0x4e22 bytes) 000359 1787361342.325087 mapread 0x002e24 thru 0x00d7af (0xa98c bytes) 000360 1787361342.326617 trunc from 0x02d254 to 0x0123b7 (0x1ae9c bytes) 000361 1787361342.347536 trunc from 0x0123b8 to 0x024b23 (0x1276c bytes) 000362 1787361342.368081 mapread 0x00cdf0 thru 0x018a88 (0xbc99 bytes) 000363 1787361342.396304 mapwrite 0x02b952 thru 0x037bb4 (0xc263 bytes) 000364 1787361342.497807 write 0x017eb7 thru 0x0214c9 (0x9613 bytes) 000367 1787361342.508823 read 0x02a170 thru 0x031f25 (0x7db6 bytes) 000370 1787361342.526257 MIRROR_RESYNC 2 mirrors 000371 1787361343.633953 mapwrite 0x01e0a1 thru 0x029121 (0xb081 bytes) 000375 1787361344.042743 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000378 1787361344.234113 read 0x01b2fc thru 0x027530 (0xc235 bytes) 000380 1787361344.249054 write 0x033f9c thru 0x03f981 (0xb9e6 bytes) 000386 1787361344.259912 read 0x013177 thru 0x01e9bc (0xb846 bytes) 000387 1787361344.272924 read 0x004ae5 thru 0x00feec (0xb408 bytes) 000389 1787361344.274467 mapread 0x028b48 thru 0x032cfa (0xa1b3 bytes) 000390 1787361344.276525 mapread 0x00ef16 thru 0x01b1fb (0xc2e6 bytes) 000393 1787361344.279777 mapwrite 0x023ad8 thru 0x032ddb (0xf304 bytes) 000394 1787361344.344001 mapread 0x01617a thru 0x025284 (0xf10b bytes) 000395 1787361344.346721 mapread 0x01cc10 thru 0x0291f9 (0xc5ea bytes) 000396 1787361344.347741 mapread 0x0351a4 thru 0x03f981 (0xa7de bytes) 000397 1787361344.349605 trunc from 0x03f982 to 0x0073ea (0x38597 bytes) 000405 1787361344.383476 trunc from 0x0073eb to 0x02a496 (0x230ac bytes) 000407 1787361344.404038 write 0x033b12 thru 0x037034 (0x3523 bytes) 000408 1787361344.406564 trunc from 0x037035 to 0x020d8c (0x162a8 bytes) 000410 1787361344.432077 read 0x009ffd thru 0x00c729 (0x272d bytes) 000411 1787361344.444485 trunc from 0x020d8d to 0x02fa66 (0xecda bytes) 000412 1787361344.468486 MIRROR_EXTEND to 2 mirrors 000413 1787361345.236044 write 0x018a6e thru 0x020907 (0x7e9a bytes) 000414 1787361345.311852 trunc from 0x02fa67 to 0x00af50 (0x24b16 bytes) 000418 1787361345.417695 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000421 1787361345.593016 MIRROR_EXTEND to 2 mirrors 000422 1787361346.534826 trunc from 0x00af51 to 0x0203d3 (0x15483 bytes) 000423 1787361346.668825 write 0x000ada thru 0x0081ca (0x76f1 bytes) 000426 1787361346.687816 trunc from 0x0203d4 to 0x027012 (0x6c3f bytes) 000427 1787361346.734907 mapread 0x004e65 thru 0x00d6de (0x887a bytes) 000429 1787361346.771452 mapwrite 0x00be04 thru 0x0133dc (0x75d9 bytes) 000430 1787361346.828960 trunc from 0x027013 to 0x038af7 (0x11ae5 bytes) 000431 1787361346.843856 read 0x01917f thru 0x01d7d6 (0x4658 bytes) 000433 1787361346.851098 write 0x03a0b2 thru 0x03b704 (0x1653 bytes) 000434 1787361346.855822 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b705): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b705): Mirror 2: 0x3764793f (stale) CRC-32 checksum value for chunk [0, 0x3b705): Mirror 1: 0xeadaac04 Mirror 3: 0xeadaac04 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b705: skipped 000436 1787361348.422979 write 0x00425e thru 0x00eed2 (0xac75 bytes) 000437 1787361348.554680 mapwrite 0x03edef thru 0x03ffff (0x1211 bytes) 000439 1787361348.731216 mapwrite 0x02f26d thru 0x037573 (0x8307 bytes) 000440 1787361348.827356 write 0x0255e2 thru 0x02be99 (0x68b8 bytes) 000441 1787361348.834521 read 0x013171 thru 0x02197c (0xe80c bytes) 000446 1787361348.852104 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xefdcdfb6 (stale) Mirror 3: 0xef7e9dc8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6e2b0913 Mirror 4: 0x6e2b0913 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000448 1787361350.572283 trunc from 0x040000 to 0x004d73 (0x3b28c bytes) 000453 1787361350.686301 read 0x0007be thru 0x004d73 (0x45b6 bytes) 000454 1787361350.695719 mapread 0x00390c thru 0x004d73 (0x1468 bytes) 000455 1787361350.701775 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4d74): Mirror 2: 0x36adc3df (stale) Mirror 4: 0x1240628c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4d74): Mirror 2: 0x36adc3df (stale) Mirror 4: 0x1240628c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4d74: skipped 000460 1787361351.089093 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4d74): Mirror 2: 0x36adc3df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4d74): Mirror 2: 0x36adc3df (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4d74: skipped 000462 1787361351.517101 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000463 1787361351.605221 trunc from 0x004d74 to 0x02e2e5 (0x29572 bytes) 000464 1787361351.628401 write 0x02095d thru 0x022f46 (0x25ea bytes) 000465 1787361351.637075 read 0x01f88c thru 0x020ea7 (0x161c bytes) 000468 1787361351.643856 mapwrite 0x0149c1 thru 0x019129 (0x4769 bytes) 000469 1787361351.708112 write 0x019c97 thru 0x0249bd (0xad27 bytes) 000471 1787361351.721984 write 0x02f436 thru 0x039f8e (0xab59 bytes) 000472 1787361351.728808 read 0x012bca thru 0x01dcc4 (0xb0fb bytes) 000473 1787361351.729728 trunc from 0x039f8f to 0x01ede0 (0x1b1ae bytes) 000474 1787361351.749783 mapread 0x015f0d thru 0x01bbdc (0x5cd0 bytes) 000476 1787361351.751942 mapread 0x00fa62 thru 0x013290 (0x382f bytes) 000477 1787361351.753841 read 0x010a0e thru 0x01ede0 (0xe3d3 bytes) 000478 1787361351.755112 read 0x001f08 thru 0x009fe2 (0x80db bytes) 000480 1787361351.764689 mapwrite 0x027fe8 thru 0x03235d (0xa376 bytes) 000483 1787361351.849070 mapwrite 0x03f2f6 thru 0x03ffff (0xd0a bytes) 000487 1787361351.927708 mapwrite 0x022530 thru 0x029e27 (0x78f8 bytes) 000488 1787361351.972432 read 0x032153 thru 0x035156 (0x3004 bytes) 000489 1787361351.972963 trunc from 0x040000 to 0x01d8fa (0x22705 bytes) 000490 1787361351.995036 write 0x0140b4 thru 0x0207ca (0xc717 bytes) 000493 1787361351.999727 mapread 0x0156ca thru 0x016aa0 (0x13d7 bytes) 000494 1787361352.000668 trunc from 0x0207cb to 0x005b52 (0x1ac78 bytes) 000496 1787361352.027448 trunc from 0x005b53 to 0x00a23c (0x46ea bytes) 000499 1787361352.065235 read 0x00a1af thru 0x00a23c (0x8e bytes) 000502 1787361352.076573 read 0x006699 thru 0x00a23c (0x3ba4 bytes) 000503 1787361352.077217 trunc from 0x00a23d to 0x02d35b (0x2311f bytes) 000505 1787361352.098995 mapread 0x01db78 thru 0x02d35b (0xf7e4 bytes) 000507 1787361352.110913 read 0x02bcfd thru 0x02d35b (0x165f bytes) 000508 1787361352.111705 write 0x0376b0 thru 0x03ffff (0x8950 bytes) 000509 1787361352.113815 read 0x01d974 thru 0x0224af (0x4b3c bytes) 000512 1787361352.116452 write 0x023be9 thru 0x02b075 (0x748d bytes) 000513 1787361352.118027 read 0x001756 thru 0x0047de (0x3089 bytes) 000514 1787361352.119372 trunc from 0x040000 to 0x006ec9 (0x39136 bytes) 000515 1787361352.136667 mapread 0x005786 thru 0x006ec9 (0x1744 bytes) 000516 1787361352.137703 mapread 0x0061f8 thru 0x006ec9 (0xcd2 bytes) 000517 1787361352.139685 trunc from 0x006eca to 0x023551 (0x1c688 bytes) 000521 1787361352.160727 mapwrite 0x009863 thru 0x010eb0 (0x764e bytes) 000522 1787361352.269950 mapwrite 0x00a15e thru 0x015358 (0xb1fb bytes) 000523 1787361352.310591 write 0x018639 thru 0x02171e (0x90e6 bytes) 000524 1787361352.311951 read 0x0105f2 thru 0x014769 (0x4178 bytes) 000525 1787361352.312489 read 0x00b096 thru 0x01147e (0x63e9 bytes) 000526 1787361352.312870 trunc from 0x023552 to 0x01eabe (0x4a93 bytes) 000527 1787361352.331564 mapwrite 0x009504 thru 0x00b5ff (0x20fc bytes) 000531 1787361352.383089 mapwrite 0x03adca thru 0x03ffff (0x5236 bytes) 000532 1787361352.470363 mapwrite 0x018c62 thru 0x0270e8 (0xe487 bytes) 000534 1787361352.524059 trunc from 0x040000 to 0x03a0b8 (0x5f47 bytes) 000535 1787361352.540984 write 0x01df71 thru 0x024029 (0x60b9 bytes) 000536 1787361352.542093 write 0x014aef thru 0x01f081 (0xa593 bytes) 000538 1787361352.543975 write 0x027b1a thru 0x02bc07 (0x40ee bytes) 000539 1787361352.548043 mapwrite 0x024741 thru 0x028837 (0x40f7 bytes) 000540 1787361352.602363 mapread 0x026f13 thru 0x03208d (0xb17b bytes) 000545 1787361352.605579 mapread 0x02a86e thru 0x032b27 (0x82ba bytes) 000546 1787361352.606531 mapread 0x019416 thru 0x02158f (0x817a bytes) 000547 1787361352.607693 mapwrite 0x023448 thru 0x02c5f6 (0x91af bytes) 000548 1787361352.693483 mapread 0x0270b2 thru 0x029cb8 (0x2c07 bytes) 000550 1787361352.694861 mapwrite 0x03f7b1 thru 0x03ffff (0x84f bytes) 000551 1787361352.799351 trunc from 0x040000 to 0x02824c (0x17db3 bytes) 000553 1787361352.814914 mapread 0x01daa7 thru 0x02219c (0x46f6 bytes) 000554 1787361352.815583 write 0x023b7e thru 0x026961 (0x2de4 bytes) 000557 1787361352.826492 write 0x0052fe thru 0x009c9c (0x499f bytes) 000559 1787361352.832537 write 0x03158e thru 0x0347a7 (0x321a bytes) 000560 1787361352.835053 write 0x02c640 thru 0x02ec48 (0x2609 bytes) 000563 1787361352.851690 mapread 0x0190e1 thru 0x022460 (0x9380 bytes) 000564 1787361352.852991 write 0x02f1d5 thru 0x03be70 (0xcc9c bytes) 000565 1787361352.859537 mapread 0x001a36 thru 0x005cd7 (0x42a2 bytes) 000568 1787361352.862748 trunc from 0x03be71 to 0x02d58e (0xe8e2 bytes) 000569 1787361352.882919 mapread 0x01558c thru 0x019240 (0x3cb5 bytes) 000571 1787361352.885873 trunc from 0x02d58f to 0x03b115 (0xdb87 bytes) 000572 1787361352.904697 write 0x01d4a2 thru 0x0264a2 (0x9001 bytes) 000575 1787361352.908066 mapread 0x028710 thru 0x030d27 (0x8618 bytes) 000581 1787361352.947743 trunc from 0x03b116 to 0x00a2b0 (0x30e65 bytes) 000583 1787361352.972653 read 0x0036c1 thru 0x00a2b0 (0x6bf0 bytes) 000586 1787361352.974903 mapread 0x00994c thru 0x00a2b0 (0x965 bytes) 000587 1787361352.976472 mapwrite 0x01179d thru 0x014e23 (0x3687 bytes) 000588 1787361353.059796 write 0x001588 thru 0x008ec9 (0x7942 bytes) 000589 1787361353.062576 read 0x001a7d thru 0x00823c (0x67c0 bytes) 000590 1787361353.064523 mapwrite 0x00d985 thru 0x0178b2 (0x9f2e bytes) 000591 1787361353.161794 write 0x03703a thru 0x0383bb (0x1382 bytes) 000594 1787361353.165284 mapread 0x014018 thru 0x014ae6 (0xacf bytes) 000595 1787361353.167493 mapwrite 0x03aa98 thru 0x03ffff (0x5568 bytes) 000597 1787361353.256399 mapread 0x035644 thru 0x03fbd1 (0xa58e bytes) 000598 1787361353.258513 mapread 0x014db6 thru 0x017d68 (0x2fb3 bytes) 000600 1787361353.266788 MIRROR_EXTEND to 2 mirrors 000601 1787361354.148092 trunc from 0x040000 to 0x01195f (0x2e6a0 bytes) 000602 1787361354.248032 mapwrite 0x01d4ee thru 0x029434 (0xbf47 bytes) 000604 1787361354.369976 write 0x02ef9c thru 0x03e8a4 (0xf909 bytes) 000605 1787361354.376907 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000607 1787361354.519587 mapwrite 0x022142 thru 0x02c8cd (0xa78c bytes) 000612 1787361354.579895 mapwrite 0x005a67 thru 0x00f702 (0x9c9c bytes) 000613 1787361354.632995 mapread 0x030a7e thru 0x03250e (0x1a91 bytes) 000614 1787361354.650644 MIRROR_EXTEND to 2 mirrors 000616 1787361355.420186 MIRROR_EXTEND to 3 mirrors 000617 1787361356.051941 trunc from 0x03e8a5 to 0x01bff8 (0x228ac bytes) 000618 1787361356.161723 mapread 0x011b40 thru 0x01bff8 (0xa4b9 bytes) 000620 1787361356.185949 read 0x006215 thru 0x006e82 (0xc6e bytes) 000622 1787361356.192887 trunc from 0x01bff9 to 0x03a844 (0x1e84c bytes) 000624 1787361356.226578 trunc from 0x03a845 to 0x026a6f (0x13dd5 bytes) 000628 1787361356.244964 trunc from 0x026a70 to 0x0009d2 (0x2609d bytes) 000629 1787361356.269761 read 0x000569 thru 0x0009d2 (0x46a bytes) 000630 1787361356.277951 read 0x000102 thru 0x0009d2 (0x8d1 bytes) 000631 1787361356.278613 mapread 0x000228 thru 0x0009d2 (0x7ab bytes) 000632 1787361356.279972 write 0x0237ed thru 0x028d58 (0x556c bytes) 000633 1787361356.281623 trunc from 0x028d59 to 0x03c800 (0x13aa8 bytes) 000634 1787361356.301324 mapwrite 0x00deee thru 0x013973 (0x5a86 bytes) 000636 1787361356.361697 mapwrite 0x020329 thru 0x02df59 (0xdc31 bytes) 000640 1787361356.418829 mapwrite 0x0144c1 thru 0x0176b1 (0x31f1 bytes) 000642 1787361356.461407 read 0x00a8ab thru 0x013f90 (0x96e6 bytes) 000645 1787361356.462962 mapwrite 0x03e5b9 thru 0x03ffff (0x1a47 bytes) 000648 1787361356.534348 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8720be11 (stale) Mirror 3: 0x8720be11 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdd7b637b Mirror 4: 0xdd7b637b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000651 1787361357.634985 mapread 0x02f9de thru 0x03b788 (0xbdab bytes) 000652 1787361357.704949 mapread 0x02d9ff thru 0x03c752 (0xed54 bytes) 000654 1787361357.734578 mapwrite 0x01b640 thru 0x02505b (0x9a1c bytes) 000656 1787361357.987362 mapwrite 0x01c028 thru 0x02994e (0xd927 bytes) 000657 1787361358.053061 write 0x01175b thru 0x0134c4 (0x1d6a bytes) 000659 1787361358.073667 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8720be11 (stale) Mirror 4: 0xdd7b637b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8720be11 (stale) Mirror 4: 0xdd7b637b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000660 1787361358.415864 write 0x03256c thru 0x0358c1 (0x3356 bytes) 000661 1787361358.435671 MIRROR_RESYNC 3 mirrors 000662 1787361360.037704 mapread 0x00b0b9 thru 0x016183 (0xb0cb bytes) 000664 1787361360.066673 write 0x03d36a thru 0x03ffff (0x2c96 bytes) 000665 1787361360.148412 mapread 0x035ed4 thru 0x03ffff (0xa12c bytes) 000666 1787361360.167020 mapwrite 0x03d0a0 thru 0x03ffff (0x2f60 bytes) 000667 1787361360.218786 write 0x00c83b thru 0x016cdb (0xa4a1 bytes) 000669 1787361360.229873 read 0x018fa5 thru 0x01f491 (0x64ed bytes) 000670 1787361360.240619 write 0x02292f thru 0x02e95c (0xc02e bytes) 000673 1787361360.248412 mapwrite 0x000120 thru 0x0016e4 (0x15c5 bytes) 000674 1787361360.394395 read 0x03d868 thru 0x03ffff (0x2798 bytes) 000675 1787361360.395398 read 0x00fb95 thru 0x012dc1 (0x322d bytes) 000676 1787361360.405460 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6f32eb02 (stale) Mirror 4: 0x6f32eb02 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b19adc Mirror 5: 0x4b19adc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000677 1787361361.891909 write 0x014b3e thru 0x023db3 (0xf276 bytes) 000681 1787361361.973844 read 0x00848d thru 0x0084ec (0x60 bytes) 000682 1787361361.979449 write 0x032101 thru 0x034160 (0x2060 bytes) 000684 1787361361.990840 mapwrite 0x02dee2 thru 0x03c1dc (0xe2fb bytes) 000686 1787361362.067914 mapread 0x03fb2b thru 0x03ffff (0x4d5 bytes) 000688 1787361362.077640 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6f32eb02 (stale) Mirror 4: 0x6f32eb02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6f32eb02 (stale) Mirror 4: 0x6f32eb02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000692 1787361362.431370 read 0x029836 thru 0x02a931 (0x10fc bytes) 000693 1787361362.438570 trunc from 0x040000 to 0x01b570 (0x24a8f bytes) 000695 1787361362.471789 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1b571): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b571): Mirror 3: 0x640f77d3 (stale) Mirror 4: 0x640f77d3 (stale) CRC-32 checksum value for chunk [0, 0x1b571): Mirror 1: 0xca8423d5 Mirror 5: 0xca8423d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b571: skipped 000696 1787361363.295444 trunc from 0x01b571 to 0x016f4e (0x4622 bytes) 000697 1787361363.467468 MIRROR_RESYNC 4 mirrors 000698 1787361365.171426 mapwrite 0x01fd00 thru 0x029658 (0x9959 bytes) 000699 1787361365.355101 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29659): Mirror 3: 0xf5bb1b90 (stale) Mirror 4: 0xf5bb1b90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29659): Mirror 3: 0xf5bb1b90 (stale) Mirror 4: 0xf5bb1b90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29659: skipped 000700 1787361365.695467 mapwrite 0x023f8d thru 0x0276fd (0x3771 bytes) 000701 1787361365.757989 mapwrite 0x0073ca thru 0x0165a6 (0xf1dd bytes) 000702 1787361365.851919 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29659): Mirror 3: 0xf5bb1b90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29659): Mirror 3: 0xf5bb1b90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29659: skipped 000704 1787361366.101084 read 0x011da1 thru 0x01b6c2 (0x9922 bytes) 000705 1787361366.110440 write 0x03f7e9 thru 0x03ffff (0x817 bytes) 000707 1787361366.112673 trunc from 0x040000 to 0x0141b3 (0x2be4c bytes) 000708 1787361366.133894 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x141b4): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x141b4): Mirror 3: 0x19d1f367 (stale) CRC-32 checksum value for chunk [0, 0x141b4): Mirror 1: 0x1859f65e Mirror 4: 0x1859f65e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x141b4: skipped 000709 1787361366.973058 mapwrite 0x002232 thru 0x004859 (0x2628 bytes) 000710 1787361367.155377 trunc from 0x0141b4 to 0x03ba60 (0x278ad bytes) 000711 1787361367.190737 mapread 0x03999e thru 0x03aaa5 (0x1108 bytes) 000713 1787361367.200449 write 0x023178 thru 0x031fc1 (0xee4a bytes) 000717 1787361367.211039 write 0x01f962 thru 0x02f139 (0xf7d8 bytes) 000718 1787361367.216421 trunc from 0x03ba61 to 0x00b471 (0x305ef bytes) 000719 1787361367.254663 MIRROR_RESYNC 3 mirrors 000720 1787361368.436880 mapread 0x00acec thru 0x00b471 (0x786 bytes) 000723 1787361368.450810 mapwrite 0x01efd5 thru 0x02841a (0x9446 bytes) 000724 1787361368.612504 read 0x01383e thru 0x020784 (0xcf47 bytes) 000725 1787361368.619324 trunc from 0x02841b to 0x002586 (0x25e94 bytes) 000728 1787361368.654929 read 0x000039 thru 0x002586 (0x254e bytes) 000732 1787361368.667813 read 0x001098 thru 0x002586 (0x14ef bytes) 000733 1787361368.669142 write 0x03de69 thru 0x03ffff (0x2197 bytes) 000734 1787361368.670269 write 0x020fb8 thru 0x026ed6 (0x5f1f bytes) 000737 1787361368.683756 trunc from 0x040000 to 0x03118c (0xee73 bytes) 000738 1787361368.701030 trunc from 0x03118d to 0x008dce (0x283be bytes) 000739 1787361368.734234 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x8dcf): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8dcf): Mirror 3: 0xabbf2455 (stale) Mirror 4: 0xabbf2455 (stale) CRC-32 checksum value for chunk [0, 0x8dcf): Mirror 1: 0xa8ed067 Mirror 5: 0xa8ed067 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8dcf: skipped 000743 1787361370.050655 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8dcf): Mirror 3: 0xabbf2455 (stale) Mirror 4: 0xabbf2455 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8dcf): Mirror 3: 0xabbf2455 (stale) Mirror 4: 0xabbf2455 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8dcf: skipped 000744 1787361370.381104 MIRROR_RESYNC 3 mirrors 000745 1787361371.796281 trunc from 0x008dcf to 0x034e4a (0x2c07c bytes) 000749 1787361371.897830 read 0x02cafd thru 0x0341b3 (0x76b7 bytes) 000750 1787361371.914326 MIRROR_RESYNC 3 mirrors 000751 1787361373.577776 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000754 1787361374.052528 mapwrite 0x00bf96 thru 0x01346d (0x74d8 bytes) 000755 1787361374.441680 read 0x0325e5 thru 0x034e4a (0x2866 bytes) 000756 1787361374.468295 trunc from 0x034e4b to 0x012123 (0x22d27 bytes) 000757 1787361374.513750 mapread 0x009f0e thru 0x00e97b (0x4a6e bytes) 000758 1787361374.517489 read 0x00e7f4 thru 0x011954 (0x3161 bytes) 000759 1787361374.519492 read 0x01093d thru 0x012123 (0x17e7 bytes) 000760 1787361374.524752 MIRROR_RESYNC 2 mirrors 000761 1787361375.833797 write 0x01c3eb thru 0x028a00 (0xc616 bytes) 000762 1787361375.891957 trunc from 0x028a01 to 0x03a6c5 (0x11cc5 bytes) 000763 1787361375.924273 mapread 0x0236c0 thru 0x023b1a (0x45b bytes) 000764 1787361375.925669 mapwrite 0x00da29 thru 0x00f51c (0x1af4 bytes) 000765 1787361376.031284 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a6c6): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a6c6): Mirror 4: 0x811a51ce (stale) CRC-32 checksum value for chunk [0, 0x3a6c6): Mirror 1: 0xec280d1b Mirror 5: 0xec280d1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a6c6: skipped 000767 1787361378.352839 read 0x02c61e thru 0x039b70 (0xd553 bytes) 000768 1787361378.374329 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000770 1787361378.840771 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000772 1787361379.069834 read 0x0281e4 thru 0x0378b1 (0xf6ce bytes) 000773 1787361379.090351 mapread 0x012bb4 thru 0x02015c (0xd5a9 bytes) 000775 1787361379.157447 write 0x00743f thru 0x008fa0 (0x1b62 bytes) 000777 1787361379.204479 read 0x017aef thru 0x01dac5 (0x5fd7 bytes) 000779 1787361379.236986 mapread 0x010e14 thru 0x01274e (0x193b bytes) 000780 1787361379.237988 mapwrite 0x03fe61 thru 0x03ffff (0x19f bytes) 000781 1787361379.408333 mapread 0x018a3c thru 0x0283f1 (0xf9b6 bytes) 000782 1787361379.409399 write 0x0010e3 thru 0x00bde9 (0xad07 bytes) 000784 1787361379.412448 read 0x0057d6 thru 0x012b2f (0xd35a bytes) 000786 1787361379.413947 mapwrite 0x00c07e thru 0x00f906 (0x3889 bytes) 000787 1787361379.549955 read 0x006e2f thru 0x013900 (0xcad2 bytes) 000789 1787361379.556320 MIRROR_EXTEND to 2 mirrors 000790 1787361380.986826 read 0x02a779 thru 0x02a77e (0x6 bytes) 000793 1787361381.037521 trunc from 0x040000 to 0x01372c (0x2c8d3 bytes) 000794 1787361381.315798 read 0x00466d thru 0x00df99 (0x992d bytes) 000796 1787361381.348387 write 0x032f48 thru 0x038993 (0x5a4c bytes) 000798 1787361381.379349 MIRROR_RESYNC 2 mirrors 000799 1787361382.881391 trunc from 0x038994 to 0x01114d (0x27846 bytes) 000800 1787361383.016339 MIRROR_RESYNC 2 mirrors 000801 1787361384.175908 mapread 0x009049 thru 0x00da24 (0x49dc bytes) 000802 1787361384.220264 mapread 0x0072c7 thru 0x0096c0 (0x23fa bytes) 000807 1787361384.230882 MIRROR_EXTEND to 3 mirrors 000809 1787361385.075505 trunc from 0x01114e to 0x012cb0 (0x1b63 bytes) 000810 1787361385.188889 read 0x011ff6 thru 0x012cb0 (0xcbb bytes) 000813 1787361385.201345 MIRROR_RESYNC 3 mirrors 000814 1787361386.286558 write 0x00f2c1 thru 0x0136e9 (0x4429 bytes) 000815 1787361386.396042 mapread 0x004604 thru 0x0076c2 (0x30bf bytes) 000816 1787361386.413418 read 0x010af1 thru 0x0136e9 (0x2bf9 bytes) 000818 1787361386.414626 read 0x007088 thru 0x0083cb (0x1344 bytes) 000819 1787361386.415671 mapwrite 0x038641 thru 0x03ffff (0x79bf bytes) 000821 1787361386.537666 mapwrite 0x02dff5 thru 0x030aae (0x2aba bytes) 000822 1787361386.590283 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd7b9ed61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd7b9ed61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000824 1787361386.929027 write 0x038cd9 thru 0x03a99f (0x1cc7 bytes) 000827 1787361386.943315 write 0x01b074 thru 0x023527 (0x84b4 bytes) 000830 1787361386.956117 mapwrite 0x01bab5 thru 0x02623c (0xa788 bytes) 000831 1787361387.162133 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) 000832 1787361387.438957 mapwrite 0x027668 thru 0x0316cf (0xa068 bytes) 000833 1787361387.528091 mapwrite 0x005150 thru 0x0064cb (0x137c bytes) 000835 1787361387.595031 write 0x038fa5 thru 0x03eeb1 (0x5f0d bytes) 000837 1787361387.608083 write 0x0038b9 thru 0x00f197 (0xb8df bytes) 000843 1787361387.623099 read 0x00b649 thru 0x014953 (0x930b bytes) 000844 1787361387.635420 mapread 0x023e5c thru 0x02cd0c (0x8eb1 bytes) 000848 1787361387.641075 read 0x008c00 thru 0x013ffb (0xb3fc bytes) 000853 1787361387.645741 write 0x003087 thru 0x00ace5 (0x7c5f bytes) 000854 1787361387.647637 mapwrite 0x01c74d thru 0x01d4e0 (0xd94 bytes) 000857 1787361387.763419 trunc from 0x040000 to 0x01db4f (0x224b0 bytes) 000858 1787361387.784648 mapread 0x01b4fc thru 0x01db4f (0x2654 bytes) 000859 1787361387.790768 write 0x0200c9 thru 0x025223 (0x515b bytes) 000860 1787361387.794341 mapread 0x007614 thru 0x00b964 (0x4351 bytes) 000862 1787361387.796731 trunc from 0x025224 to 0x02045b (0x4dc8 bytes) 000864 1787361387.817653 read 0x00b75e thru 0x0163af (0xac52 bytes) 000866 1787361387.821744 mapread 0x008852 thru 0x017cd4 (0xf483 bytes) 000867 1787361387.823094 mapwrite 0x038f82 thru 0x03ffff (0x707e bytes) 000868 1787361387.933769 write 0x009e67 thru 0x016c8d (0xce27 bytes) 000869 1787361387.934996 mapread 0x022ef2 thru 0x0289d7 (0x5ae6 bytes) 000872 1787361387.938772 trunc from 0x040000 to 0x033a45 (0xc5ba bytes) 000876 1787361387.967758 read 0x0252aa thru 0x027615 (0x236c bytes) 000881 1787361387.972249 read 0x0130c9 thru 0x01b6d6 (0x860e bytes) 000882 1787361387.973365 write 0x0287dd thru 0x037889 (0xf0ad bytes) 000883 1787361387.976305 trunc from 0x03788a to 0x0392f9 (0x1a70 bytes) 000884 1787361387.997762 mapwrite 0x02d44b thru 0x038d94 (0xb94a bytes) 000885 1787361388.073733 read 0x01bf63 thru 0x022f80 (0x701e bytes) 000886 1787361388.074605 trunc from 0x0392fa to 0x026e20 (0x124d9 bytes) 000887 1787361388.091000 trunc from 0x026e21 to 0x026f9a (0x17a bytes) 000888 1787361388.103621 mapread 0x01758a thru 0x0251fe (0xdc75 bytes) 000889 1787361388.104469 mapread 0x01f8be thru 0x02094b (0x108e bytes) 000890 1787361388.108339 MIRROR_EXTEND to 2 mirrors 000891 1787361389.370705 trunc from 0x026f9b to 0x03825b (0x112c1 bytes) 000892 1787361389.483066 trunc from 0x03825c to 0x01f40d (0x18e4e bytes) 000893 1787361389.513218 trunc from 0x01f40e to 0x01f57f (0x172 bytes) 000894 1787361389.552582 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1f580): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f580): Mirror 2: 0x14baf04 (stale) CRC-32 checksum value for chunk [0, 0x1f580): Mirror 1: 0x14baf04 Mirror 3: 0x14baf04 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f580: skipped 000895 1787361390.602668 write 0x02c4d4 thru 0x037c24 (0xb751 bytes) 000897 1787361390.771116 mapwrite 0x021c9c thru 0x026ace (0x4e33 bytes) 000898 1787361390.896415 mapwrite 0x028c86 thru 0x035525 (0xc8a0 bytes) 000900 1787361391.021675 trunc from 0x037c25 to 0x027c3b (0xffe9 bytes) 000901 1787361391.059002 mapread 0x0230e1 thru 0x027c3b (0x4b5b bytes) 000903 1787361391.061535 write 0x00d8e5 thru 0x016659 (0x8d75 bytes) 000904 1787361391.082680 write 0x020a1a thru 0x0307ef (0xfdd6 bytes) 000905 1787361391.087054 read 0x01c62f thru 0x02a3db (0xddad bytes) 000906 1787361391.088107 write 0x017a70 thru 0x02080b (0x8d9c bytes) 000908 1787361391.100567 trunc from 0x0307f0 to 0x02acbd (0x5b32 bytes) 000909 1787361391.125084 trunc from 0x02acbe to 0x001d24 (0x28f99 bytes) 000910 1787361391.159624 mapread 0x001911 thru 0x001d24 (0x414 bytes) 000911 1787361391.167865 mapread 0x0004be thru 0x001d24 (0x1867 bytes) 000913 1787361391.181726 mapread 0x000d24 thru 0x001d24 (0x1001 bytes) 000914 1787361391.183786 write 0x014355 thru 0x01c00a (0x7cb6 bytes) 000916 1787361391.187890 trunc from 0x01c00b to 0x025194 (0x918a bytes) 000919 1787361391.226122 write 0x005526 thru 0x006782 (0x125d bytes) 000920 1787361391.240611 mapwrite 0x02e570 thru 0x02e74f (0x1e0 bytes) 000921 1787361391.342695 MIRROR_RESYNC 3 mirrors 000922 1787361392.755610 mapwrite 0x00e163 thru 0x016a3b (0x88d9 bytes) 000923 1787361392.968711 MIRROR_RESYNC 3 mirrors 000926 1787361394.473200 write 0x021453 thru 0x02ff81 (0xeb2f bytes) 000927 1787361394.555037 write 0x00c268 thru 0x012f05 (0x6c9e bytes) 000928 1787361394.564449 trunc from 0x02ff82 to 0x038227 (0x82a6 bytes) 000929 1787361394.593111 MIRROR_RESYNC 3 mirrors 000930 1787361396.736979 read 0x033d5c thru 0x038227 (0x44cc bytes) 000931 1787361396.752713 trunc from 0x038228 to 0x03767e (0xba9 bytes) 000932 1787361396.889846 read 0x01859e thru 0x018d1c (0x77f bytes) 000934 1787361396.905948 write 0x00581e thru 0x006bf5 (0x13d8 bytes) 000937 1787361396.909940 read 0x02b66e thru 0x033000 (0x7993 bytes) 000941 1787361396.915197 mapwrite 0x025002 thru 0x0253a7 (0x3a6 bytes) 000943 1787361396.973562 mapread 0x010e94 thru 0x01bb2d (0xac9a bytes) 000950 1787361396.983234 trunc from 0x03767f to 0x02eea6 (0x87d8 bytes) 000951 1787361397.030320 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2eea7): Mirror 3: 0x3726745c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eea7): Mirror 3: 0x3726745c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eea7: skipped 000954 1787361397.422490 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2eea7): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eea7): Mirror 3: 0x3726745c (stale) CRC-32 checksum value for chunk [0, 0x2eea7): Mirror 1: 0x2b40a03c Mirror 4: 0x2b40a03c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eea7: skipped 000956 1787361398.650055 trunc from 0x02eea7 to 0x0128a8 (0x1c5fe bytes) 000958 1787361398.729767 read 0x00e9e0 thru 0x0128a8 (0x3ec9 bytes) 000959 1787361398.738238 read 0x00fe57 thru 0x0128a8 (0x2a52 bytes) 000960 1787361398.738859 trunc from 0x0128a9 to 0x00b97a (0x6f2e bytes) 000961 1787361398.757563 mapread 0x003c24 thru 0x00b97a (0x7d57 bytes) 000962 1787361398.767848 read 0x0078f7 thru 0x00b97a (0x4084 bytes) 000963 1787361398.768492 trunc from 0x00b97b to 0x02fb44 (0x241ca bytes) 000966 1787361398.790078 trunc from 0x02fb45 to 0x02f239 (0x90b bytes) 000967 1787361398.808358 trunc from 0x02f23a to 0x02df68 (0x12d1 bytes) 000968 1787361398.828811 write 0x03e728 thru 0x03ffff (0x18d8 bytes) 000970 1787361398.831436 mapread 0x021b17 thru 0x022d65 (0x124f bytes) 000972 1787361398.845617 mapread 0x03db97 thru 0x03ffff (0x2469 bytes) 000973 1787361398.846639 mapwrite 0x022899 thru 0x0311c8 (0xe930 bytes) 000976 1787361398.901940 trunc from 0x040000 to 0x025db3 (0x1a24c bytes) 000981 1787361398.925423 read 0x01c276 thru 0x0209cb (0x4756 bytes) 000982 1787361398.930434 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25db4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25db4): Mirror 3: 0xa2c6c472 (stale) Mirror 4: 0x87c349de (stale) CRC-32 checksum value for chunk [0, 0x25db4): Mirror 1: 0x7082bf4b Mirror 5: 0x7082bf4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25db4: skipped 000985 1787361400.292380 write 0x0276f3 thru 0x02da5c (0x636a bytes) 000986 1787361400.445797 MIRROR_RESYNC 4 mirrors 000988 1787361402.450018 mapwrite 0x00afde thru 0x00ec06 (0x3c29 bytes) 000992 1787361402.665851 MIRROR_RESYNC 4 mirrors 000994 1787361404.657316 mapwrite 0x017653 thru 0x0269bd (0xf36b bytes) 000995 1787361404.933985 write 0x03c7a5 thru 0x03ffff (0x385b bytes) 000996 1787361404.935393 write 0x001c7f thru 0x0066d3 (0x4a55 bytes) 000997 1787361404.953073 mapread 0x029f41 thru 0x032d75 (0x8e35 bytes) 000998 1787361404.970044 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4a85bd06 (stale) Mirror 4: 0x4a85bd06 (stale) Mirror 5: 0x4a85bd06 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfce09e69 Mirror 6: 0xfce09e69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000999 1787361406.643965 mapread 0x01f378 thru 0x029d2e (0xa9b7 bytes) All operations completed A-OK! PASS 70a (111s) == sanity-flr test 71: check mirror extend parent fid ==== 21:16:53 (1787361413) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 21:16:57 (1787361417) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.367059 s, 5.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161785 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 21:17:08 (1787361428) Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre2 Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1524128 .. Extending file size to 7397888 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7014528 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7922752 .. Extending file size to 7769152 .. Extending file size to 8209728 .. Extending file size to 7947360 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8553120 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8357888 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9272800 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8864160 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 57427 57428 57429 57431 57433 192.168.202.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg204-client.virtnet /mnt/lustre2 (opts:) 192.168.202.104@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg204-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:18:30 (1787361510) Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre2 Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4204064 .. Extending file size to 7837056 .. Extending file size to 7945408 .. Extending file size to 7294848 .. Extending file size to 7453120 .. Extending file size to 8194656 .. Extending file size to 8701888 .. Extending file size to 7840736 .. Extending file size to 8128416 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9009728 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8537568 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 58970 58971 58974 58976 58977 192.168.202.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg204-client.virtnet /mnt/lustre2 (opts:) 192.168.202.104@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg204-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:19:48 (1787361588) Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0994237 s, 21.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.397061 s, 5.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x7d108126 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg204-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:20:09 (1787361609) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0856623 s, 24.5 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 21:20:17 (1787361617) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x304:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x305:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2e8:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:20:23 (1787361623) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128884 s, 16.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ea:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:20:32 (1787361632) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.286074 s, 7.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:20:41 (1787361641) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.483019 s, 4.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x309:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ee:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:21:16 (1787361676) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg204-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120178 s, 8.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30a:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30b:0x0] } PASS 204d (61s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:22:17 (1787361737) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg204-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.434983 s, 4.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (63s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:23:21 (1787361801) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg204-server mds-ost sync done. Creating new pool test_204f oleg204-server: Pool lustre.test_204f created Adding targets to pool oleg204-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg204-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274354 s, 7.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg204-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg204-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg204-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:24:42 (1787361882) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:24:50 (1787361890) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:24:57 (1787361897) oleg204-server: Pool lustre.test_206 created Adding targets to pool oleg204-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg204-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg204-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg204-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg204-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:25:26 (1787361926) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972445 s, 10.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347913 s, 30.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x129a9ec9 Mirror 3: 0x129a9ec9 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:25:36 (1787361936) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:25:39 (1787361939) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:25:42 (1787361942) Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.164464 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1170000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x18e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x18e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x18e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1000000 Read this (4 bytes): BBBB 192.168.202.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg204-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:26:20 (1787361980) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0428689 s, 191 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:26:27 (1787361987) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464675 s, 22.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x215 lmm_fid: [0x200000404:0x215:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834453 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-21 21:26:29.000000000 -0400 Modify: 2026-08-21 21:26:29.000000000 -0400 Change: 2026-08-21 21:26:30.000000000 -0400 Birth: 2026-08-21 21:26:29.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:26:35 (1787361995) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:26:39 (1787361999) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.318994 s, 128 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0347902 s, 118 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 21 21:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x217:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 21 21:26 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:26:48 (1787362008) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:26:57 (1787362017) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1865 sec =========== 21:27:06 (1787362026) === sanity-flr: start cleanup 21:27:08 (1787362028) === === sanity-flr: finish cleanup 21:27:16 (1787362036) ===