-----============= acceptance-small: sanity-flr ============----- Tue Aug 25 07:51:23 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg441-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 07:51:46 (1787658706) === oleg441-client.virtnet: executing check_config_client /mnt/lustre oleg441-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg441-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa01ed1866800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa01ed1866800.idle_timeout=debug disable quota as required oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 07:52:30 (1787658750) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8477] == sanity-flr test 0a: lfs mirror create with -N option == 07:52:33 (1787658753) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:52:52 (1787658772) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:52:56 (1787658776) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:52:59 (1787658779) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:53:29 (1787658809) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:53:33 (1787658813) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 07:53:37 (1787658817) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa01ed1866800.stats=clear osc.lustre-OST0001-osc-ffffa01ed1866800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:53:48 (1787658828) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:54:02 (1787658842) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:54:12 (1787658852) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:54:15 (1787658855) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:54:24 (1787658864) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 07:54:33 (1787658873) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:54:41 (1787658881) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:54:50 (1787658890) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:54:58 (1787658898) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:55:09 (1787658909) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:55:18 (1787658918) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.395382 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708591 s, 14.8 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:55:32 (1787658932) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.408996 s, 28.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:55:41 (1787658941) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:55:53 (1787658953) Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa01ed1866800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa01ed1866800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg441-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa01ed1866800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa01ed1866800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg441-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (136s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:58:09 (1787659089) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.502152 s, 12.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa01ed1866800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa01ed1866800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (49s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:58:58 (1787659138) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa01ed1866800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa01ed1866800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa01ed1866800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa01ed1866800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg441-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (89s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:00:27 (1787659227) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.16739 s, 10.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 25 08:00 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.103s user 0m0.001s sys 0m0.174s Stopping /mnt/lustre-ost1 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa01ed1866800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa01ed1866800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.407s user 0m0.000s sys 0m0.171s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg441-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg441-server oleg441-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa01ed1866800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa01ed1866800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.561s user 0m0.001s sys 0m0.212s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg441-server: oleg441-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg441-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg441-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa01ed1866800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa01ed1866800.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:02:11 (1787659331) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:02:14 (1787659334) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:02:18 (1787659338) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 08:02:21 (1787659341) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 08:02:28 (1787659348) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.549279 s, 22.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:02:41 (1787659361) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637188 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357263 s, 29.4 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:02:48 (1787659368) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:02:56 (1787659376) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00466969 s, 219 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00107539 s, 952 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0057981 s, 177 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0141485 s, 72.4 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 08:03:09 (1787659389) create test files with size 15752485 .. 7bb56bcb28da4970045c09554b588f53 - 3cc6220fd70d7722f97b8870bcc2a46e - bbb92303a50a1c41bbf5fe17c24428d9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg441-client.virtnet /mnt/lustre (opts:) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 08:03:35 (1787659415) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:03:54 (1787659434) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:04:01 (1787659441) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699746 s, 15.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407636 s, 25.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 08:04:10 (1787659450) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165446 s, 12.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.35915 s, 11.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.243755 s, 8.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 08:04:29 (1787659469) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 08:04:32 (1787659472) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:04:36 (1787659476) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252283 s, 41.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 08:04:44 (1787659484) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.302194 s, 10.4 MB/s PASS 44a (13s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:04:57 (1787659497) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:05:09 (1787659509) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.305938 s, 34.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg441-client.virtnet /mnt/lustre (opts:) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:05:23 (1787659523) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa01ed1665800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.26988 s, 38.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg441-client.virtnet /mnt/lustre (opts:) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffffa01ef2f3a800.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:05:44 (1787659544) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:06:25 (1787659585) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (47s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:07:12 (1787659632) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 08:07:16 (1787659636) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.265525 s, 11.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.201659 s, 15.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5ef165d7 (nosync,stale) Mirror 2: 0x5ef165d7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: e7698c30f33924a07f241e06fe7a9de4 - mirror 1 checksum: 1fe6854c590574678decc79c4320fab4 - mirror 2 checksum: e7698c30f33924a07f241e06fe7a9de4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-25T08:07:19-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-25T08:07:19-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e7698c30f33924a07f241e06fe7a9de4 - mirror 1 checksum: 1fe6854c590574678decc79c4320fab4 - mirror 2 checksum: 1fe6854c590574678decc79c4320fab4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-25T08:07:19-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-25T08:07:19-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:07:30 (1787659650) Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server 08:07:38 (1787659658) shut down facet: mds1 facet_host: oleg441-server facet_failover_host: oleg441-server Failover mds1 to oleg441-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 08:08:14 (1787659694) targets are mounted 08:08:14 (1787659694) facet_failover done oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306932 s, 34.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd lmm_fid: [0x200000405:0xd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:08:33 (1787659713) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:08:37 (1787659717) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:08:41 (1787659721) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0170452 s, 961 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:08:50 (1787659730) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:08:54 (1787659734) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0157217 s, 261 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0593681 s, 69.0 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:09:04 (1787659744) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:09:32 (1787659772) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:10:04 (1787659804) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xd:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:10:30 (1787659830) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:10:57 (1787659857) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 08:11:06 (1787659866) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 56476 mirror split pid 56477 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 56476 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 56477 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 08:12:14 (1787659934) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 9936 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000002 1787659936.526413 mapwrite 0x002856 thru 0x003138 (0x8e3 bytes) 000004 1787659936.667730 mapread 0x001f1e thru 0x003138 (0x121b bytes) 000005 1787659936.681056 read 0x0015c2 thru 0x003138 (0x1b77 bytes) truncating to largest ever: 0x3a859 000006 1787659936.682875 trunc from 0x003139 to 0x03a858 (0x37720 bytes) 000007 1787659936.720584 mapread 0x024f89 thru 0x02620f (0x1287 bytes) 000009 1787659936.741769 mapread 0x01e503 thru 0x02b2d3 (0xcdd1 bytes) 000011 1787659936.745987 mapread 0x011067 thru 0x01f12d (0xe0c7 bytes) 000014 1787659936.791126 MIRROR_EXTEND to 2 mirrors 000016 1787659937.999338 mapread 0x031931 thru 0x033a5a (0x212a bytes) 000017 1787659938.069888 mapread 0x030520 thru 0x03a858 (0xa339 bytes) truncating to largest ever: 0x3e355 000018 1787659938.108054 trunc from 0x03a859 to 0x03e354 (0x3afc bytes) 000020 1787659938.238333 write 0x0135a7 thru 0x01f8a3 (0xc2fd bytes) 000021 1787659938.256565 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) 000022 1787659938.446802 write 0x02ea13 thru 0x030480 (0x1a6e bytes) 000025 1787659938.464393 write 0x03027e thru 0x03ba66 (0xb7e9 bytes) 000027 1787659938.481264 trunc from 0x03e355 to 0x0268a5 (0x17aaf bytes) 000028 1787659938.508945 write 0x03b3f7 thru 0x03ffff (0x4c09 bytes) 000029 1787659938.511479 read 0x02d92f thru 0x0387c7 (0xae99 bytes) truncating to largest ever: 0x3e955 000030 1787659938.524182 trunc from 0x040000 to 0x03e954 (0x16ab bytes) 000031 1787659938.548184 mapread 0x02d82c thru 0x0386d5 (0xaeaa bytes) 000033 1787659938.553039 mapwrite 0x022091 thru 0x0279fa (0x596a bytes) 000038 1787659938.673592 mapread 0x01468e thru 0x01dce6 (0x9659 bytes) 000041 1787659938.689030 mapread 0x01a69a thru 0x01f7bf (0x5126 bytes) 000042 1787659938.690977 write 0x02d79c thru 0x03b2dd (0xdb42 bytes) 000043 1787659938.693921 write 0x01863b thru 0x01c992 (0x4358 bytes) 000044 1787659938.695153 read 0x02272e thru 0x02c96f (0xa242 bytes) 000045 1787659938.697339 trunc from 0x03e955 to 0x00add7 (0x33b7d bytes) 000046 1787659938.724297 write 0x00ee04 thru 0x019bdb (0xadd8 bytes) 000048 1787659938.731510 mapread 0x00fe9b thru 0x014cd9 (0x4e3f bytes) 000049 1787659938.734298 mapwrite 0x002448 thru 0x0038cc (0x1485 bytes) 000050 1787659938.875148 mapread 0x0186f0 thru 0x019bdb (0x14ec bytes) 000052 1787659938.881499 mapwrite 0x00fc23 thru 0x01f6e9 (0xfac7 bytes) 000059 1787659939.057853 trunc from 0x01f6ea to 0x015279 (0xa470 bytes) 000061 1787659939.094097 read 0x00ae65 thru 0x00e204 (0x33a0 bytes) 000064 1787659939.099656 write 0x00e02f thru 0x0175fa (0x95cc bytes) 000066 1787659939.106048 mapread 0x0060b9 thru 0x00a74e (0x4696 bytes) 000067 1787659939.111828 write 0x0032f5 thru 0x006958 (0x3664 bytes) 000070 1787659939.122906 write 0x001d07 thru 0x004547 (0x2841 bytes) 000071 1787659939.127659 mapread 0x00fc86 thru 0x015094 (0x540f bytes) 000072 1787659939.132558 mapwrite 0x02096e thru 0x0306c1 (0xfd54 bytes) 000074 1787659939.278030 mapwrite 0x01e343 thru 0x02b37e (0xd03c bytes) 000075 1787659939.340021 mapread 0x002b5b thru 0x00b00e (0x84b4 bytes) 000076 1787659939.343303 mapread 0x005ba6 thru 0x006728 (0xb83 bytes) 000077 1787659939.345462 write 0x004eaf thru 0x0076cb (0x281d bytes) 000081 1787659939.353620 trunc from 0x0306c2 to 0x00fc81 (0x20a40 bytes) 000082 1787659939.375923 write 0x0294dc thru 0x031183 (0x7ca8 bytes) 000083 1787659939.379851 read 0x00f4e3 thru 0x01b617 (0xc135 bytes) 000084 1787659939.391777 write 0x00555e thru 0x007bf2 (0x2695 bytes) 000086 1787659939.394867 mapread 0x009e5c thru 0x015728 (0xb8cd bytes) 000087 1787659939.398642 mapread 0x00e27b thru 0x010992 (0x2718 bytes) 000088 1787659939.400237 mapwrite 0x008487 thru 0x014fc5 (0xcb3f bytes) 000089 1787659939.484595 write 0x024fc9 thru 0x031c56 (0xcc8e bytes) 000090 1787659939.494546 mapwrite 0x01a85e thru 0x0229e2 (0x8185 bytes) 000095 1787659939.600085 mapwrite 0x0215c2 thru 0x02c0f5 (0xab34 bytes) 000097 1787659939.714540 mapwrite 0x00af43 thru 0x01a311 (0xf3cf bytes) 000101 1787659939.817152 mapwrite 0x02d9ff thru 0x03703c (0x963e bytes) 000102 1787659939.920271 read 0x0229ce thru 0x02ba78 (0x90ab bytes) 000103 1787659939.923423 mapwrite 0x024158 thru 0x024872 (0x71b bytes) 000104 1787659939.966087 mapwrite 0x0397ea thru 0x03ffff (0x6816 bytes) 000105 1787659940.089843 trunc from 0x040000 to 0x00163a (0x3e9c5 bytes) 000107 1787659940.133065 mapread 0x0007e2 thru 0x00163a (0xe59 bytes) 000110 1787659940.144295 mapwrite 0x00bde7 thru 0x0179d6 (0xbbf0 bytes) 000111 1787659940.427087 mapwrite 0x034173 thru 0x03447a (0x308 bytes) 000112 1787659940.504173 mapread 0x00c5c9 thru 0x00d691 (0x10c9 bytes) 000113 1787659940.505233 mapread 0x00ea41 thru 0x01be76 (0xd436 bytes) 000114 1787659940.534129 mapwrite 0x003e07 thru 0x00b9ee (0x7be8 bytes) 000118 1787659940.657911 mapwrite 0x020c9a thru 0x023341 (0x26a8 bytes) 000119 1787659940.725970 trunc from 0x03447b to 0x02e81b (0x5c5f bytes) 000120 1787659940.747281 mapread 0x020685 thru 0x029dfb (0x9777 bytes) 000121 1787659940.779264 mapread 0x006fe7 thru 0x014e58 (0xde72 bytes) 000123 1787659940.781821 write 0x0003ea thru 0x00f6ff (0xf316 bytes) 000124 1787659940.785430 write 0x0294ce thru 0x0384ac (0xefdf bytes) 000125 1787659940.787985 trunc from 0x0384ad to 0x03c2b8 (0x3e0c bytes) truncating to largest ever: 0x3fcde 000127 1787659940.814102 trunc from 0x03c2b9 to 0x03fcdd (0x3a25 bytes) 000130 1787659940.841231 read 0x02be91 thru 0x03b694 (0xf804 bytes) 000132 1787659940.852733 mapread 0x001b7a thru 0x004435 (0x28bc bytes) 000133 1787659940.853795 trunc from 0x03fcde to 0x00dece (0x31e0f bytes) 000134 1787659940.871859 mapread 0x0024f0 thru 0x00ab2f (0x8640 bytes) 000135 1787659940.873034 read 0x003175 thru 0x00879a (0x5626 bytes) 000136 1787659940.877974 mapread 0x00c1ad thru 0x00dece (0x1d22 bytes) 000139 1787659940.881505 mapwrite 0x033b98 thru 0x03b706 (0x7b6f bytes) 000140 1787659940.978985 write 0x00bf61 thru 0x01bef3 (0xff93 bytes) 000145 1787659940.985136 mapread 0x030253 thru 0x036d52 (0x6b00 bytes) 000151 1787659940.991436 mapread 0x024a7d thru 0x026a28 (0x1fac bytes) 000152 1787659940.992597 trunc from 0x03b707 to 0x032119 (0x95ed bytes) 000153 1787659941.009787 mapwrite 0x000029 thru 0x00aa4d (0xaa25 bytes) 000155 1787659941.075563 trunc from 0x03211a to 0x01c723 (0x159f6 bytes) 000157 1787659941.092631 mapwrite 0x022c6c thru 0x028325 (0x56ba bytes) 000158 1787659941.196030 mapwrite 0x011b29 thru 0x0206f8 (0xebd0 bytes) 000160 1787659941.273109 read 0x000de3 thru 0x00ef14 (0xe132 bytes) 000162 1787659941.274386 write 0x026b06 thru 0x02e15b (0x7656 bytes) 000163 1787659941.276659 trunc from 0x02e15c to 0x030c9f (0x2b44 bytes) 000164 1787659941.297201 trunc from 0x030ca0 to 0x027aeb (0x91b4 bytes) 000169 1787659941.340033 mapread 0x008c7f thru 0x010a6e (0x7df0 bytes) 000171 1787659941.344317 mapwrite 0x035bf6 thru 0x03ae62 (0x526d bytes) 000172 1787659941.447408 write 0x02c984 thru 0x035de4 (0x9461 bytes) 000173 1787659941.462752 MIRROR_EXTEND to 2 mirrors 000174 1787659942.698801 write 0x00e471 thru 0x01483d (0x63cd bytes) 000175 1787659942.878959 mapwrite 0x0331b1 thru 0x03ffff (0xce4f bytes) 000176 1787659943.096350 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) 000177 1787659943.307245 trunc from 0x040000 to 0x02c1ff (0x13e00 bytes) 000178 1787659943.345379 write 0x03629a thru 0x03ffff (0x9d66 bytes) 000179 1787659943.346858 mapwrite 0x027afd thru 0x02f180 (0x7684 bytes) 000180 1787659943.476543 read 0x01c530 thru 0x0296ff (0xd1d0 bytes) 000186 1787659943.494566 trunc from 0x040000 to 0x01c900 (0x236ff bytes) 000188 1787659943.526905 trunc from 0x01c901 to 0x014e08 (0x7af8 bytes) 000189 1787659943.555606 read 0x013918 thru 0x014e08 (0x14f1 bytes) 000192 1787659943.569570 mapread 0x009944 thru 0x014e08 (0xb4c5 bytes) 000195 1787659943.573027 write 0x014717 thru 0x0245af (0xfe99 bytes) 000196 1787659943.575538 mapread 0x014972 thru 0x0245af (0xfc3e bytes) 000198 1787659943.595828 write 0x01ee08 thru 0x02bb06 (0xccff bytes) 000199 1787659943.597526 write 0x026056 thru 0x02d77d (0x7728 bytes) 000201 1787659943.600595 mapwrite 0x01fa58 thru 0x02dd41 (0xe2ea bytes) 000202 1787659943.699011 write 0x00063c thru 0x004ec9 (0x488e bytes) 000203 1787659943.701938 write 0x024147 thru 0x02de5b (0x9d15 bytes) 000204 1787659943.704517 read 0x01e144 thru 0x02cadd (0xe99a bytes) 000205 1787659943.705553 mapwrite 0x030749 thru 0x03a412 (0x9cca bytes) 000206 1787659943.848125 mapread 0x024e7b thru 0x02da37 (0x8bbd bytes) 000207 1787659943.849370 read 0x00439e thru 0x007d68 (0x39cb bytes) 000213 1787659943.862373 trunc from 0x03a413 to 0x0162ee (0x24124 bytes) 000214 1787659943.883720 mapread 0x0158ab thru 0x0162ee (0xa44 bytes) 000216 1787659943.886149 trunc from 0x0162ef to 0x02b860 (0x15572 bytes) 000217 1787659943.900846 mapwrite 0x00827f thru 0x00d5b7 (0x5339 bytes) 000219 1787659943.958195 write 0x01eb56 thru 0x026da7 (0x8252 bytes) 000220 1787659943.971164 mapwrite 0x01d204 thru 0x0244e2 (0x72df bytes) 000221 1787659944.032500 mapread 0x01478f thru 0x01a964 (0x61d6 bytes) 000226 1787659944.042453 mapread 0x002b1d thru 0x00ae75 (0x8359 bytes) 000229 1787659944.047404 write 0x030cc5 thru 0x03ee61 (0xe19d bytes) 000230 1787659944.055138 MIRROR_EXTEND to 2 mirrors 000232 1787659945.248498 write 0x01f8c0 thru 0x0215d4 (0x1d15 bytes) 000235 1787659945.351198 read 0x0324dc thru 0x03ee61 (0xc986 bytes) 000238 1787659945.375155 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ee62): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ee62): Mirror 2: 0x55d123b4 (stale) CRC-32 checksum value for chunk [0, 0x3ee62): Mirror 1: 0xe55de6c7 Mirror 3: 0xe55de6c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ee62: skipped 000240 1787659947.110490 mapwrite 0x022580 thru 0x02b85b (0x92dc bytes) 000242 1787659947.340584 mapread 0x023491 thru 0x02666d (0x31dd bytes) 000244 1787659947.348497 MIRROR_RESYNC 3 mirrors 000247 1787659948.819050 mapread 0x02d1ce thru 0x03ac12 (0xda45 bytes) 000250 1787659948.881086 write 0x02c3ff thru 0x032976 (0x6578 bytes) 000251 1787659949.043806 read 0x029949 thru 0x038bbb (0xf273 bytes) 000252 1787659949.068553 mapread 0x021e27 thru 0x02b0ec (0x92c6 bytes) 000256 1787659949.101160 read 0x0229c5 thru 0x0252d3 (0x290f bytes) 000257 1787659949.105398 write 0x02c831 thru 0x0359d7 (0x91a7 bytes) 000258 1787659949.108314 trunc from 0x03ee62 to 0x004945 (0x3a51c bytes) 000259 1787659949.137838 trunc from 0x004946 to 0x0237d2 (0x1ee8d bytes) 000261 1787659949.183884 write 0x00797a thru 0x009ae4 (0x216b bytes) 000263 1787659949.206830 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x237d3): Mirror 3: 0xa033a5ee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x237d3): Mirror 3: 0xa033a5ee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x237d3: skipped 000266 1787659949.698863 mapwrite 0x01e859 thru 0x02ac84 (0xc42c bytes) 000267 1787659949.884128 mapread 0x01b947 thru 0x0264dc (0xab96 bytes) 000268 1787659949.886574 mapread 0x011845 thru 0x01e0ec (0xc8a8 bytes) 000269 1787659949.900409 trunc from 0x02ac85 to 0x02f952 (0x4cce bytes) 000270 1787659949.928415 write 0x011c73 thru 0x01757d (0x590b bytes) 000271 1787659949.933181 mapwrite 0x00343f thru 0x00d124 (0x9ce6 bytes) 000273 1787659950.003882 mapwrite 0x02e7eb thru 0x03b426 (0xcc3c bytes) 000274 1787659950.102282 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) 000275 1787659950.254093 mapwrite 0x0365ac thru 0x03b858 (0x52ad bytes) 000277 1787659950.345018 write 0x00a9d6 thru 0x0123e5 (0x7a10 bytes) 000278 1787659950.359246 read 0x00648d thru 0x00ad0c (0x4880 bytes) 000279 1787659950.370832 write 0x008b56 thru 0x00b386 (0x2831 bytes) 000284 1787659950.376920 trunc from 0x03b859 to 0x024738 (0x17120 bytes) 000289 1787659950.419075 trunc from 0x024739 to 0x028c19 (0x44e1 bytes) 000290 1787659950.456962 write 0x01c330 thru 0x023bbb (0x788c bytes) 000292 1787659950.494777 mapread 0x019c7d thru 0x026650 (0xc9d4 bytes) 000293 1787659950.529544 write 0x03530c thru 0x03ffff (0xacf4 bytes) 000294 1787659950.555594 MIRROR_EXTEND to 2 mirrors 000296 1787659952.272606 mapread 0x039d30 thru 0x03ced7 (0x31a8 bytes) 000297 1787659952.309418 read 0x00bece thru 0x018327 (0xc45a bytes) 000299 1787659952.340344 MIRROR_RESYNC 2 mirrors 000300 1787659952.880806 mapread 0x024261 thru 0x02d368 (0x9108 bytes) 000301 1787659952.943476 mapwrite 0x023ac0 thru 0x02879d (0x4cde bytes) 000303 1787659953.220743 mapread 0x0122bd thru 0x01aa72 (0x87b6 bytes) 000306 1787659953.232423 mapread 0x02518a thru 0x033df2 (0xec69 bytes) 000309 1787659953.242263 trunc from 0x040000 to 0x031e24 (0xe1db bytes) 000311 1787659953.263070 mapwrite 0x00d62f thru 0x01a45d (0xce2f bytes) 000312 1787659953.323889 mapwrite 0x00cb51 thru 0x010b58 (0x4008 bytes) 000313 1787659953.366988 write 0x016215 thru 0x019309 (0x30f5 bytes) 000315 1787659953.368752 read 0x01131e thru 0x013c9f (0x2982 bytes) 000316 1787659953.370985 mapread 0x02330d thru 0x026f60 (0x3c54 bytes) 000318 1787659953.373562 mapwrite 0x035a0c thru 0x03a9b8 (0x4fad bytes) 000319 1787659953.453970 mapread 0x01fe0d thru 0x02d1ef (0xd3e3 bytes) 000320 1787659953.455598 mapread 0x02fe1e thru 0x0366a4 (0x6887 bytes) 000321 1787659953.457765 write 0x007dee thru 0x01593e (0xdb51 bytes) 000324 1787659953.466901 read 0x02c52d thru 0x0316f7 (0x51cb bytes) 000326 1787659953.469028 read 0x03133f thru 0x03a9b8 (0x967a bytes) 000328 1787659953.471268 trunc from 0x03a9b9 to 0x014ff5 (0x259c3 bytes) 000332 1787659953.497022 mapwrite 0x02d5bf thru 0x02fa8e (0x24d0 bytes) 000333 1787659953.594064 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2fa8f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa8f): Mirror 2: 0xf5d852bc (stale) CRC-32 checksum value for chunk [0, 0x2fa8f): Mirror 1: 0xd5ad4cd3 Mirror 3: 0xd5ad4cd3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa8f: skipped 000334 1787659955.023176 trunc from 0x02fa8f to 0x01c050 (0x13a3e bytes) 000335 1787659955.152774 write 0x030557 thru 0x0396fc (0x91a6 bytes) 000336 1787659955.155777 write 0x02240c thru 0x027721 (0x5316 bytes) 000337 1787659955.170590 write 0x00fcf0 thru 0x01d49e (0xd7af bytes) 000338 1787659955.236749 trunc from 0x0396fd to 0x029d5c (0xf9a0 bytes) 000341 1787659955.299216 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29d5d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29d5d): Mirror 2: 0xe8919a94 (stale) Mirror 3: 0xf7c658c7 (stale) CRC-32 checksum value for chunk [0, 0x29d5d): Mirror 1: 0x9a7bbc83 Mirror 4: 0x9a7bbc83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29d5d: skipped 000344 1787659957.136643 mapwrite 0x0243ff thru 0x02e069 (0x9c6b bytes) 000346 1787659957.389236 mapwrite 0x0117a9 thru 0x01258d (0xde5 bytes) 000347 1787659957.444784 mapwrite 0x017ba9 thru 0x026658 (0xeab0 bytes) 000348 1787659957.510992 write 0x031773 thru 0x03b86f (0xa0fd bytes) 000351 1787659957.518573 trunc from 0x03b870 to 0x00a3c6 (0x314a9 bytes) 000352 1787659957.545953 write 0x028385 thru 0x02cfd3 (0x4c4f bytes) 000354 1787659957.549586 trunc from 0x02cfd4 to 0x010188 (0x1ce4b bytes) 000355 1787659957.572933 read 0x0044cd thru 0x00f08d (0xabc1 bytes) 000356 1787659957.583467 mapwrite 0x01afc1 thru 0x02866a (0xd6aa bytes) 000358 1787659957.681357 mapread 0x00e630 thru 0x01c9b2 (0xe383 bytes) 000359 1787659957.691927 read 0x000ed1 thru 0x004b55 (0x3c85 bytes) 000361 1787659957.700124 read 0x008864 thru 0x00db39 (0x52d6 bytes) 000364 1787659957.716037 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2866b): Mirror 3: 0x4a8de3a2 (stale) Mirror 4: 0x99db4431 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2866b): Mirror 3: 0x4a8de3a2 (stale) Mirror 4: 0x99db4431 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2866b: skipped 000365 1787659958.241759 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2866b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2866b): Mirror 3: 0x4a8de3a2 (stale) Mirror 4: 0x99db4431 (stale) CRC-32 checksum value for chunk [0, 0x2866b): Mirror 1: 0x3ee9d648 Mirror 5: 0x3ee9d648 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2866b: skipped 000366 1787659959.608153 mapread 0x00468a thru 0x006ae0 (0x2457 bytes) 000367 1787659959.664852 trunc from 0x02866b to 0x03e92d (0x162c3 bytes) 000368 1787659959.836135 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e92e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e92e): Mirror 3: 0x8f920a09 (stale) Mirror 4: 0xe41b2035 (stale) Mirror 5: 0x6510c460 (stale) CRC-32 checksum value for chunk [0, 0x3e92e): Mirror 1: 0x6510c460 Mirror 6: 0x6510c460 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e92e: skipped 000371 1787659961.719587 mapwrite 0x00b0f2 thru 0x01a41e (0xf32d bytes) 000372 1787659961.947413 read 0x0276f1 thru 0x03474b (0xd05b bytes) 000373 1787659961.958185 mapread 0x02f2f1 thru 0x037b20 (0x8830 bytes) 000374 1787659961.973786 trunc from 0x03e92e to 0x016c76 (0x27cb7 bytes) 000376 1787659962.002223 write 0x02ddc6 thru 0x03c629 (0xe864 bytes) 000377 1787659962.006631 read 0x032a98 thru 0x03c629 (0x9b92 bytes) 000379 1787659962.009190 trunc from 0x03c62a to 0x02a962 (0x11cc7 bytes) 000380 1787659962.055228 mapread 0x01a64d thru 0x020a0d (0x63c1 bytes) 000384 1787659962.072502 mapread 0x00c0c2 thru 0x00f756 (0x3695 bytes) 000387 1787659962.077178 trunc from 0x02a963 to 0x02ff59 (0x55f7 bytes) 000391 1787659962.103111 trunc from 0x02ff5a to 0x00fb5d (0x203fc bytes) 000392 1787659962.123097 trunc from 0x00fb5e to 0x01cc7f (0xd122 bytes) 000393 1787659962.139067 write 0x022c30 thru 0x02ffee (0xd3bf bytes) 000394 1787659962.141829 write 0x00ae0f thru 0x00f998 (0x4b8a bytes) 000396 1787659962.144820 mapwrite 0x027051 thru 0x034b74 (0xdb24 bytes) 000398 1787659962.246714 mapwrite 0x03718a thru 0x0372ff (0x176 bytes) 000400 1787659962.345841 mapwrite 0x01f865 thru 0x0276c0 (0x7e5c bytes) 000401 1787659962.456567 MIRROR_RESYNC 5 mirrors 000402 1787659965.188281 trunc from 0x037300 to 0x013271 (0x2408e bytes) 000403 1787659965.390177 read 0x00aad6 thru 0x013271 (0x879c bytes) 000405 1787659965.437523 read 0x003684 thru 0x008870 (0x51ed bytes) 000406 1787659965.467046 read 0x010b52 thru 0x013271 (0x2720 bytes) 000408 1787659965.478521 write 0x035c6f thru 0x03ffff (0xa391 bytes) 000409 1787659965.487319 mapwrite 0x02479a thru 0x025c02 (0x1469 bytes) 000419 1787659965.716970 mapwrite 0x00737a thru 0x016d8e (0xfa15 bytes) 000422 1787659965.863549 read 0x02de17 thru 0x02e0a7 (0x291 bytes) 000424 1787659965.866718 mapwrite 0x009725 thru 0x00c58c (0x2e68 bytes) 000425 1787659965.942197 write 0x03af68 thru 0x03d6c2 (0x275b bytes) 000428 1787659965.950115 write 0x033ef9 thru 0x03725f (0x3367 bytes) 000429 1787659965.955711 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa6926c4 (stale) Mirror 4: 0xfa6926c4 (stale) Mirror 5: 0xfa6926c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa6926c4 (stale) Mirror 4: 0xfa6926c4 (stale) Mirror 5: 0xfa6926c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000430 1787659966.459167 read 0x02d7c0 thru 0x039c06 (0xc447 bytes) 000431 1787659966.471371 trunc from 0x040000 to 0x037135 (0x8eca bytes) 000432 1787659966.528131 MIRROR_RESYNC 4 mirrors 000434 1787659969.186265 MIRROR_EXTEND to 5 mirrors 000436 1787659970.670083 trunc from 0x037136 to 0x0111e3 (0x25f52 bytes) 000437 1787659970.770559 trunc from 0x0111e4 to 0x024090 (0x12ead bytes) 000438 1787659970.793283 mapwrite 0x01d9b7 thru 0x01f543 (0x1b8d bytes) 000440 1787659970.859540 read 0x0108cf thru 0x0150db (0x480d bytes) 000441 1787659970.870508 read 0x00e6dc thru 0x01a3e7 (0xbd0c bytes) 000443 1787659970.882547 trunc from 0x024091 to 0x015807 (0xe889 bytes) 000446 1787659970.905573 MIRROR_RESYNC 5 mirrors 000447 1787659972.948936 read 0x0000fc thru 0x008fa9 (0x8eae bytes) 000448 1787659972.963595 mapwrite 0x03ff84 thru 0x03ffff (0x7c bytes) 000452 1787659973.126515 write 0x009f0b thru 0x00c15d (0x2253 bytes) 000453 1787659973.187581 trunc from 0x040000 to 0x027492 (0x18b6d bytes) 000456 1787659973.263662 trunc from 0x027493 to 0x02353e (0x3f54 bytes) 000457 1787659973.296932 write 0x01658f thru 0x025635 (0xf0a7 bytes) 000458 1787659973.306575 write 0x004e49 thru 0x01204b (0xd203 bytes) 000462 1787659973.333733 trunc from 0x025636 to 0x00d259 (0x183dc bytes) 000463 1787659973.387163 read 0x0046be thru 0x00c05d (0x79a0 bytes) 000464 1787659973.388280 read 0x001a5e thru 0x00a6bc (0x8c5f bytes) 000465 1787659973.397227 mapwrite 0x0256a2 thru 0x03067e (0xafdd bytes) 000466 1787659973.577469 write 0x038492 thru 0x03ffff (0x7b6e bytes) 000467 1787659973.579951 write 0x01e15f thru 0x02ac35 (0xcad7 bytes) 000471 1787659973.585816 mapwrite 0x029b0f thru 0x02b940 (0x1e32 bytes) 000474 1787659973.688774 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x67505301 (stale) Mirror 4: 0x67505301 (stale) Mirror 5: 0x67505301 (stale) Mirror 6: 0x67505301 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5bd13fa1 Mirror 7: 0x5bd13fa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000475 1787659975.545646 mapwrite 0x02c46d thru 0x02d1a8 (0xd3c bytes) 000477 1787659976.091390 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x67505301 (stale) Mirror 4: 0x67505301 (stale) Mirror 5: 0x67505301 (stale) Mirror 6: 0x67505301 (stale) Mirror 7: 0x5bd13fa1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8080e251 Mirror 8: 0x8080e251 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000480 1787659978.447442 mapread 0x016be1 thru 0x019d59 (0x3179 bytes) 000481 1787659978.493845 read 0x024076 thru 0x02b56e (0x74f9 bytes) 000482 1787659978.515715 mapwrite 0x0046f9 thru 0x0092cc (0x4bd4 bytes) 000483 1787659978.700362 write 0x00c059 thru 0x00d52f (0x14d7 bytes) 000484 1787659978.708253 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x67505301 (stale) Mirror 4: 0x67505301 (stale) Mirror 6: 0x67505301 (stale) Mirror 7: 0x5bd13fa1 (stale) Mirror 8: 0x8080e251 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x67505301 (stale) Mirror 4: 0x67505301 (stale) Mirror 6: 0x67505301 (stale) Mirror 7: 0x5bd13fa1 (stale) Mirror 8: 0x8080e251 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000486 1787659979.383736 mapwrite 0x013253 thru 0x019366 (0x6114 bytes) 000488 1787659979.605342 mapread 0x010aeb thru 0x01dbab (0xd0c1 bytes) 000490 1787659979.614002 MIRROR_RESYNC 6 mirrors 000492 1787659983.247322 write 0x00da9c thru 0x00e92c (0xe91 bytes) 000493 1787659983.383200 mapread 0x000b04 thru 0x00503f (0x453c bytes) 000497 1787659983.435009 mapread 0x002c5f thru 0x00e076 (0xb418 bytes) 000498 1787659983.439075 mapwrite 0x014f55 thru 0x023180 (0xe22c bytes) 000499 1787659983.620623 read 0x003b5f thru 0x00ee74 (0xb316 bytes) 000501 1787659983.630159 MIRROR_RESYNC 6 mirrors 000504 1787659986.851306 write 0x005fa1 thru 0x00efe5 (0x9045 bytes) 000506 1787659986.947500 write 0x03063b thru 0x032450 (0x1e16 bytes) 000507 1787659986.962268 mapread 0x03994a thru 0x039dc4 (0x47b bytes) 000512 1787659986.973085 read 0x015808 thru 0x020acd (0xb2c6 bytes) 000513 1787659986.994564 MIRROR_RESYNC 6 mirrors 000515 1787659990.132550 mapread 0x02f2be thru 0x034425 (0x5168 bytes) 000517 1787659990.153923 trunc from 0x040000 to 0x034eca (0xb135 bytes) 000519 1787659990.242281 MIRROR_RESYNC 6 mirrors 000522 1787659992.982936 write 0x0342bf thru 0x03ffff (0xbd41 bytes) 000523 1787659993.088323 mapwrite 0x006362 thru 0x0150be (0xed5d bytes) 000524 1787659993.266605 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9c4cd9f1 (stale) Mirror 4: 0x9c4cd9f1 (stale) Mirror 6: 0x9c4cd9f1 (stale) Mirror 8: 0x9c4cd9f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9c4cd9f1 (stale) Mirror 4: 0x9c4cd9f1 (stale) Mirror 6: 0x9c4cd9f1 (stale) Mirror 8: 0x9c4cd9f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000525 1787659994.129438 read 0x02ca16 thru 0x039721 (0xcd0c bytes) 000527 1787659994.156923 read 0x03665a thru 0x03e0a5 (0x7a4c bytes) 000528 1787659994.202353 trunc from 0x040000 to 0x03383b (0xc7c4 bytes) 000529 1787659994.247328 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3383c): Mirror 3: 0x6ba4ae86 (stale) Mirror 4: 0x6ba4ae86 (stale) Mirror 6: 0x6ba4ae86 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3383c): Mirror 3: 0x6ba4ae86 (stale) Mirror 4: 0x6ba4ae86 (stale) Mirror 6: 0x6ba4ae86 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3383c: skipped 000530 1787659994.723817 trunc from 0x03383c to 0x015f9c (0x1d89f bytes) 000531 1787659994.757880 write 0x0222e4 thru 0x02a013 (0x7d30 bytes) 000532 1787659994.759410 mapread 0x00ac7b thru 0x00d15c (0x24e2 bytes) 000534 1787659994.808459 MIRROR_RESYNC 4 mirrors 000535 1787659997.102736 mapwrite 0x023895 thru 0x028c80 (0x53ec bytes) 000537 1787659997.347801 mapread 0x02266d thru 0x025452 (0x2de6 bytes) 000540 1787659997.353749 write 0x000a87 thru 0x004d82 (0x42fc bytes) 000541 1787659997.363942 mapwrite 0x001824 thru 0x01113d (0xf91a bytes) 000542 1787659997.538394 write 0x00918f thru 0x015f19 (0xcd8b bytes) 000543 1787659997.539833 mapwrite 0x038bd5 thru 0x03ffff (0x742b bytes) 000545 1787659997.724348 trunc from 0x040000 to 0x0267e9 (0x19816 bytes) 000546 1787659997.781575 read 0x01e3d2 thru 0x01f0a8 (0xcd7 bytes) 000550 1787659997.786631 mapread 0x001e77 thru 0x00a7a1 (0x892b bytes) 000552 1787659997.791006 write 0x02b9d1 thru 0x03a686 (0xecb6 bytes) 000553 1787659997.799400 read 0x01f05e thru 0x0207a0 (0x1743 bytes) 000556 1787659997.807320 trunc from 0x03a687 to 0x008961 (0x31d25 bytes) 000560 1787659997.854605 MIRROR_RESYNC 4 mirrors 000561 1787659999.066169 mapread 0x0011d4 thru 0x008961 (0x778e bytes) 000562 1787659999.097486 write 0x020d79 thru 0x02fffe (0xf286 bytes) 000563 1787659999.177602 mapwrite 0x007201 thru 0x009655 (0x2455 bytes) 000564 1787659999.253994 trunc from 0x02ffff to 0x0154a1 (0x1ab5d bytes) 000565 1787659999.280547 mapwrite 0x03ee8f thru 0x03ffff (0x1171 bytes) 000566 1787659999.374194 trunc from 0x040000 to 0x02593c (0x1a6c3 bytes) 000567 1787659999.391888 read 0x011dd4 thru 0x01b278 (0x94a5 bytes) 000571 1787659999.405074 mapwrite 0x012511 thru 0x01902e (0x6b1e bytes) 000573 1787659999.469171 read 0x02398c thru 0x02593c (0x1fb1 bytes) 000576 1787659999.488379 write 0x01eff2 thru 0x021d4c (0x2d5b bytes) 000577 1787659999.511035 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2593d): Mirror 3: 0x7dec2d34 (stale) Mirror 4: 0x7dec2d34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2593d): Mirror 3: 0x7dec2d34 (stale) Mirror 4: 0x7dec2d34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2593d: skipped 000579 1787659999.979951 mapread 0x023888 thru 0x02593c (0x20b5 bytes) 000581 1787660000.006785 mapwrite 0x01bdb3 thru 0x02aaf8 (0xed46 bytes) 000584 1787660000.121243 mapread 0x0029d6 thru 0x0118c3 (0xeeee bytes) 000585 1787660000.155100 MIRROR_RESYNC 3 mirrors 000586 1787660001.596203 read 0x004d0d thru 0x007d4a (0x303e bytes) 000588 1787660001.606554 MIRROR_EXTEND to 4 mirrors 000589 1787660003.184091 mapread 0x028f29 thru 0x02aaf8 (0x1bd0 bytes) 000590 1787660003.217104 read 0x01937e thru 0x025ea9 (0xcb2c bytes) 000592 1787660003.229381 mapwrite 0x001b94 thru 0x00d14c (0xb5b9 bytes) 000593 1787660003.730866 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2aaf9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aaf9): Mirror 3: 0xf09b6899 (stale) Mirror 4: 0xf09b6899 (stale) Mirror 5: 0xf09b6899 (stale) CRC-32 checksum value for chunk [0, 0x2aaf9): Mirror 1: 0xc0fa2ca0 Mirror 6: 0xc0fa2ca0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aaf9: skipped 000594 1787660005.148155 write 0x02e8eb thru 0x039ee1 (0xb5f7 bytes) 000596 1787660005.367166 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x39ee2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39ee2): Mirror 3: 0x5a3255f5 (stale) Mirror 4: 0x5a3255f5 (stale) Mirror 5: 0x5a3255f5 (stale) Mirror 6: 0x7f7e0161 (stale) CRC-32 checksum value for chunk [0, 0x39ee2): Mirror 1: 0x8ceede6d Mirror 7: 0x8ceede6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39ee2: skipped 000597 1787660007.571007 write 0x01d66c thru 0x01db0e (0x4a3 bytes) 000600 1787660007.728512 read 0x011c57 thru 0x01682f (0x4bd9 bytes) 000601 1787660007.748641 MIRROR_RESYNC 6 mirrors 000602 1787660011.027094 write 0x0310cb thru 0x03ffff (0xef35 bytes) 000603 1787660011.100988 trunc from 0x040000 to 0x028193 (0x17e6c bytes) 000604 1787660011.133891 MIRROR_RESYNC 6 mirrors 000605 1787660013.901093 MIRROR_RESYNC 6 mirrors 000606 1787660015.056062 mapwrite 0x0091ad thru 0x00f6e6 (0x653a bytes) 000610 1787660015.326741 mapwrite 0x008fe5 thru 0x00a6e0 (0x16fc bytes) 000614 1787660015.446701 read 0x00887a thru 0x0130f6 (0xa87d bytes) 000616 1787660015.466821 trunc from 0x028194 to 0x0217b3 (0x69e0 bytes) 000617 1787660015.510971 write 0x006641 thru 0x00d7f1 (0x71b1 bytes) 000619 1787660015.525200 MIRROR_RESYNC 6 mirrors 000621 1787660019.477709 trunc from 0x0217b4 to 0x01ae79 (0x693a bytes) 000622 1787660019.636489 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ae7a): Mirror 3: 0xa9bcd7e9 (stale) Mirror 4: 0xa9bcd7e9 (stale) Mirror 5: 0xa9bcd7e9 (stale) Mirror 6: 0xa9bcd7e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ae7a): Mirror 3: 0xa9bcd7e9 (stale) Mirror 4: 0xa9bcd7e9 (stale) Mirror 5: 0xa9bcd7e9 (stale) Mirror 6: 0xa9bcd7e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ae7a: skipped 000624 1787660020.203085 write 0x005cc9 thru 0x009987 (0x3cbf bytes) 000625 1787660020.219421 read 0x0049cf thru 0x014034 (0xf666 bytes) 000626 1787660020.238967 write 0x024431 thru 0x033863 (0xf433 bytes) 000627 1787660020.242717 trunc from 0x033864 to 0x03e434 (0xabd1 bytes) 000630 1787660020.274791 write 0x0295c1 thru 0x037578 (0xdfb8 bytes) 000632 1787660020.291172 write 0x019532 thru 0x026fc9 (0xda98 bytes) 000633 1787660020.307757 read 0x01795a thru 0x019098 (0x173f bytes) 000634 1787660020.330774 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e435): Mirror 4: 0xa16e04ed (stale) Mirror 5: 0xa16e04ed (stale) Mirror 6: 0xa16e04ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e435): Mirror 4: 0xa16e04ed (stale) Mirror 5: 0xa16e04ed (stale) Mirror 6: 0xa16e04ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e435: skipped 000635 1787660020.883404 mapread 0x00382d thru 0x0049b9 (0x118d bytes) 000637 1787660020.896458 mapread 0x01b60f thru 0x02a60d (0xefff bytes) 000638 1787660020.960477 MIRROR_RESYNC 4 mirrors 000639 1787660023.750373 trunc from 0x03e435 to 0x0207fa (0x1dc3a bytes) 000640 1787660023.854770 read 0x0037fa thru 0x0099e7 (0x61ee bytes) 000642 1787660023.868861 trunc from 0x0207fb to 0x000453 (0x203a7 bytes) 000644 1787660023.891814 mapwrite 0x012778 thru 0x016bff (0x4488 bytes) 000646 1787660023.992337 read 0x008fc1 thru 0x00c120 (0x3160 bytes) 000647 1787660024.005342 read 0x016aaa thru 0x016bff (0x156 bytes) 000651 1787660024.008401 trunc from 0x016c00 to 0x0073e9 (0xf816 bytes) 000653 1787660024.028178 MIRROR_RESYNC 4 mirrors 000654 1787660025.669618 MIRROR_EXTEND to 5 mirrors 000655 1787660026.647113 MIRROR_EXTEND to 6 mirrors 000656 1787660027.938358 mapread 0x0035ed thru 0x006c15 (0x3629 bytes) 000657 1787660028.019278 read 0x0031e6 thru 0x0073e9 (0x4204 bytes) 000658 1787660028.019995 trunc from 0x0073ea to 0x0014d1 (0x5f18 bytes) 000661 1787660028.397243 mapread 0x0007de thru 0x0014d1 (0xcf4 bytes) 000664 1787660028.428925 mapwrite 0x00f071 thru 0x01d81f (0xe7af bytes) 000665 1787660028.732372 trunc from 0x01d820 to 0x01e3a6 (0xb87 bytes) 000666 1787660028.784281 mapwrite 0x00e53c thru 0x00ea37 (0x4fc bytes) 000670 1787660028.937545 write 0x014119 thru 0x017d4f (0x3c37 bytes) 000674 1787660028.949488 write 0x01949d thru 0x01c407 (0x2f6b bytes) 000675 1787660028.953009 write 0x013bc8 thru 0x01e307 (0xa740 bytes) 000676 1787660028.979449 mapwrite 0x01fdcf thru 0x02f529 (0xf75b bytes) 000677 1787660029.165350 mapwrite 0x039fb9 thru 0x03ffff (0x6047 bytes) 000678 1787660029.392557 trunc from 0x040000 to 0x0143df (0x2bc20 bytes) 000681 1787660029.452639 mapread 0x013dc8 thru 0x0143df (0x618 bytes) 000683 1787660029.485852 mapwrite 0x018e6b thru 0x02402e (0xb1c4 bytes) 000684 1787660029.670239 trunc from 0x02402f to 0x007abf (0x1c56f bytes) 000685 1787660029.710946 mapread 0x00277e thru 0x00549d (0x2d20 bytes) 000687 1787660029.750961 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x7ac0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7ac0): Mirror 4: 0x4d845ea8 (stale) Mirror 5: 0x4d845ea8 (stale) Mirror 6: 0x4d845ea8 (stale) Mirror 7: 0x4d845ea8 (stale) Mirror 8: 0x4d845ea8 (stale) CRC-32 checksum value for chunk [0, 0x7ac0): Mirror 1: 0x4d845ea8 Mirror 9: 0x4d845ea8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7ac0: skipped 000688 1787660031.877168 mapwrite 0x02a0bc thru 0x02cbde (0x2b23 bytes) 000689 1787660032.181302 write 0x01e4f5 thru 0x024c6a (0x6776 bytes) 000690 1787660032.207736 mapwrite 0x018dc8 thru 0x0245ad (0xb7e6 bytes) 000692 1787660032.434272 trunc from 0x02cbdf to 0x02cded (0x20f bytes) 000693 1787660032.502909 mapwrite 0x02ee47 thru 0x0386d4 (0x988e bytes) 000696 1787660032.662916 trunc from 0x0386d5 to 0x000dc6 (0x3790e bytes) 000697 1787660032.692942 trunc from 0x000dc7 to 0x01986f (0x18aa9 bytes) 000698 1787660032.725297 trunc from 0x019870 to 0x032c14 (0x193a5 bytes) 000699 1787660032.746030 mapwrite 0x0259e5 thru 0x02de90 (0x84ac bytes) 000701 1787660032.854089 write 0x00825f thru 0x016299 (0xe03b bytes) 000702 1787660032.874840 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32c15): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32c15): Mirror 4: 0x44a2bd18 (stale) Mirror 5: 0x44a2bd18 (stale) Mirror 6: 0x44a2bd18 (stale) Mirror 7: 0x44a2bd18 (stale) Mirror 8: 0x44a2bd18 (stale) Mirror 9: 0x44a2bd18 (stale) CRC-32 checksum value for chunk [0, 0x32c15): Mirror 1: 0x588e16b4 Mirror 10: 0x588e16b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32c15: skipped 000703 1787660034.866906 mapwrite 0x02055f thru 0x0205fb (0x9d bytes) 000705 1787660035.064744 mapread 0x011954 thru 0x015aa5 (0x4152 bytes) 000707 1787660035.076346 trunc from 0x032c15 to 0x024e75 (0xdd9f bytes) 000709 1787660035.105248 mapwrite 0x010a1d thru 0x02055e (0xfb42 bytes) 000711 1787660035.192845 trunc from 0x024e76 to 0x01a65e (0xa817 bytes) 000712 1787660035.237236 mapread 0x010cc3 thru 0x012412 (0x1750 bytes) 000713 1787660035.257157 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a65f): Mirror 4: 0xcb40eb5d (stale) Mirror 5: 0xcb40eb5d (stale) Mirror 7: 0xcb40eb5d (stale) Mirror 8: 0xcb40eb5d (stale) Mirror 9: 0xcb40eb5d (stale) Mirror 10: 0xfde8b13a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a65f): Mirror 4: 0xcb40eb5d (stale) Mirror 5: 0xcb40eb5d (stale) Mirror 7: 0xcb40eb5d (stale) Mirror 8: 0xcb40eb5d (stale) Mirror 9: 0xcb40eb5d (stale) Mirror 10: 0xfde8b13a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a65f: skipped 000714 1787660035.965186 MIRROR_RESYNC 7 mirrors 000715 1787660038.357371 mapwrite 0x034145 thru 0x03b331 (0x71ed bytes) 000716 1787660038.539762 write 0x03e7bf thru 0x03ffff (0x1841 bytes) 000718 1787660038.543320 write 0x025e85 thru 0x02fc13 (0x9d8f bytes) 000720 1787660038.553933 write 0x03de2e thru 0x03ffff (0x21d2 bytes) 000721 1787660038.561146 trunc from 0x040000 to 0x030851 (0xf7ae bytes) 000722 1787660038.586358 mapwrite 0x02ed6e thru 0x03e892 (0xfb25 bytes) 000724 1787660038.712282 mapread 0x006bfb thru 0x00af7d (0x4383 bytes) 000725 1787660038.728189 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3e893): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e893): Mirror 4: 0xe51c61fb (stale) Mirror 5: 0xe51c61fb (stale) Mirror 7: 0xe51c61fb (stale) Mirror 8: 0xe51c61fb (stale) Mirror 9: 0xe51c61fb (stale) Mirror 10: 0xe51c61fb (stale) CRC-32 checksum value for chunk [0, 0x3e893): Mirror 1: 0x502fe3a7 Mirror 11: 0x502fe3a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e893: skipped 000726 1787660040.357881 write 0x03d01c thru 0x03d55f (0x544 bytes) 000727 1787660040.633224 mapwrite 0x0132ad thru 0x013690 (0x3e4 bytes) 000728 1787660040.766472 mapwrite 0x00e8c7 thru 0x0172f6 (0x8a30 bytes) 000729 1787660040.896947 mapread 0x010ccf thru 0x01fc90 (0xefc2 bytes) 000730 1787660040.932195 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e893): Mirror 5: 0xe51c61fb (stale) Mirror 7: 0xe51c61fb (stale) Mirror 8: 0xe51c61fb (stale) Mirror 9: 0xe51c61fb (stale) Mirror 10: 0xe51c61fb (stale) Mirror 11: 0x502fe3a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e893): Mirror 5: 0xe51c61fb (stale) Mirror 7: 0xe51c61fb (stale) Mirror 8: 0xe51c61fb (stale) Mirror 9: 0xe51c61fb (stale) Mirror 10: 0xe51c61fb (stale) Mirror 11: 0x502fe3a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e893: skipped 000731 1787660041.514217 write 0x0166fd thru 0x025205 (0xeb09 bytes) 000733 1787660041.528937 mapread 0x01ece1 thru 0x028b5a (0x9e7a bytes) 000734 1787660041.552415 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3e893): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e893): Mirror 5: 0xe51c61fb (stale) Mirror 7: 0xe51c61fb (stale) Mirror 8: 0xe51c61fb (stale) Mirror 9: 0xe51c61fb (stale) Mirror 10: 0xe51c61fb (stale) Mirror 11: 0x502fe3a7 (stale) CRC-32 checksum value for chunk [0, 0x3e893): Mirror 1: 0xaa3dc02c Mirror 12: 0xaa3dc02c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e893: skipped 000736 1787660042.964086 read 0x0331b3 thru 0x03d934 (0xa782 bytes) 000737 1787660043.002655 MIRROR_RESYNC 8 mirrors 000738 1787660046.179000 write 0x023e6d thru 0x025ff5 (0x2189 bytes) 000739 1787660046.359459 mapwrite 0x027cb2 thru 0x0293c1 (0x1710 bytes) 000740 1787660046.527842 mapread 0x034e9d thru 0x036fd8 (0x213c bytes) 000742 1787660046.574376 write 0x0307b6 thru 0x0364be (0x5d09 bytes) 000743 1787660046.623991 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e893): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e893): Mirror 5: 0xaa3dc02c (stale) Mirror 7: 0xaa3dc02c (stale) Mirror 8: 0xaa3dc02c (stale) Mirror 9: 0xaa3dc02c (stale) Mirror 10: 0xaa3dc02c (stale) Mirror 11: 0xaa3dc02c (stale) Mirror 12: 0xaa3dc02c (stale) CRC-32 checksum value for chunk [0, 0x3e893): Mirror 1: 0x6ad508bc Mirror 13: 0x6ad508bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e893: skipped 000744 1787660048.503856 trunc from 0x03e893 to 0x032423 (0xc46f bytes) 000745 1787660048.615919 trunc from 0x032424 to 0x01dfc8 (0x1445b bytes) 000747 1787660048.642090 mapwrite 0x003aba thru 0x012cf2 (0xf239 bytes) 000748 1787660048.756365 mapread 0x01b113 thru 0x01dfc8 (0x2eb6 bytes) 000749 1787660048.775184 mapread 0x014108 thru 0x014247 (0x140 bytes) 000751 1787660048.779774 mapwrite 0x0042d6 thru 0x00c864 (0x858f bytes) 000753 1787660048.879910 trunc from 0x01dfc9 to 0x00cd00 (0x112c8 bytes) 000755 1787660048.929088 mapread 0x0037c6 thru 0x00cd00 (0x953b bytes) 000756 1787660048.931168 mapwrite 0x02b1de thru 0x03b1b8 (0xffdb bytes) 000757 1787660049.151825 write 0x00d41f thru 0x01b568 (0xe14a bytes) 000758 1787660049.180711 mapread 0x00076b thru 0x002344 (0x1bda bytes) 000759 1787660049.193438 write 0x0014ae thru 0x00e130 (0xcc83 bytes) 000762 1787660049.212590 mapread 0x0146e7 thru 0x01a083 (0x599d bytes) 000765 1787660049.224997 trunc from 0x03b1b9 to 0x0214ff (0x19cb9 bytes) 000767 1787660049.260799 trunc from 0x021500 to 0x0092bd (0x18242 bytes) 000768 1787660049.306924 MIRROR_RESYNC 9 mirrors 000770 1787660051.888282 mapwrite 0x0009ac thru 0x0057d9 (0x4e2e bytes) 000772 1787660052.120097 read 0x0028f2 thru 0x0092bd (0x69cc bytes) 000774 1787660052.123026 mapwrite 0x028811 thru 0x02fdb9 (0x75a9 bytes) 000777 1787660052.245924 read 0x01a928 thru 0x01c442 (0x1b1b bytes) 000778 1787660052.272080 mapread 0x000032 thru 0x001d81 (0x1d50 bytes) 000779 1787660052.279497 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2fdba): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fdba): Mirror 5: 0x6ab04e2e (stale) Mirror 7: 0x6ab04e2e (stale) Mirror 8: 0x6ab04e2e (stale) Mirror 9: 0x6ab04e2e (stale) Mirror 10: 0x6ab04e2e (stale) Mirror 11: 0x6ab04e2e (stale) Mirror 12: 0x6ab04e2e (stale) Mirror 13: 0x6ab04e2e (stale) CRC-32 checksum value for chunk [0, 0x2fdba): Mirror 1: 0x6b99bf98 Mirror 14: 0x6b99bf98 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fdba: skipped 000781 1787660053.501168 mapread 0x022bd0 thru 0x02f129 (0xc55a bytes) 000782 1787660053.562608 read 0x019b76 thru 0x01d57b (0x3a06 bytes) 000783 1787660053.596959 read 0x0271d7 thru 0x02fdb9 (0x8be3 bytes) 000786 1787660053.625967 read 0x003f98 thru 0x00b5a0 (0x7609 bytes) 000787 1787660053.646011 mapwrite 0x02697f thru 0x030345 (0x99c7 bytes) 000789 1787660054.051638 mapread 0x01d5a7 thru 0x026784 (0x91de bytes) 000792 1787660054.151320 mapread 0x0296a4 thru 0x030345 (0x6ca2 bytes) 000793 1787660054.153927 mapwrite 0x01def5 thru 0x02938f (0xb49b bytes) 000794 1787660054.269272 write 0x02f31a thru 0x032e00 (0x3ae7 bytes) 000795 1787660054.271841 write 0x03d106 thru 0x03ffff (0x2efa bytes) 000796 1787660054.273489 mapwrite 0x01d7df thru 0x01e089 (0x8ab bytes) 000800 1787660054.391247 write 0x0363a7 thru 0x03ffff (0x9c59 bytes) 000801 1787660054.414966 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5dabbdcd (stale) Mirror 7: 0x5dabbdcd (stale) Mirror 8: 0x5dabbdcd (stale) Mirror 10: 0x5dabbdcd (stale) Mirror 11: 0x5dabbdcd (stale) Mirror 12: 0x5dabbdcd (stale) Mirror 13: 0x5dabbdcd (stale) Mirror 14: 0xc25c1947 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5dabbdcd (stale) Mirror 7: 0x5dabbdcd (stale) Mirror 8: 0x5dabbdcd (stale) Mirror 10: 0x5dabbdcd (stale) Mirror 11: 0x5dabbdcd (stale) Mirror 12: 0x5dabbdcd (stale) Mirror 13: 0x5dabbdcd (stale) Mirror 14: 0xc25c1947 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000802 1787660055.028149 read 0x00eb9f thru 0x01653f (0x79a1 bytes) 000803 1787660055.040402 read 0x016eb4 thru 0x01d819 (0x6966 bytes) 000804 1787660055.056999 mapwrite 0x0280c4 thru 0x034793 (0xc6d0 bytes) 000808 1787660055.128094 mapread 0x02d92b thru 0x031984 (0x405a bytes) 000809 1787660055.130586 read 0x01b409 thru 0x021a7d (0x6675 bytes) 000810 1787660055.137465 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5dabbdcd (stale) Mirror 7: 0x5dabbdcd (stale) Mirror 8: 0x5dabbdcd (stale) Mirror 10: 0x5dabbdcd (stale) Mirror 11: 0x5dabbdcd (stale) Mirror 12: 0x5dabbdcd (stale) Mirror 13: 0x5dabbdcd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x5dabbdcd (stale) Mirror 7: 0x5dabbdcd (stale) Mirror 8: 0x5dabbdcd (stale) Mirror 10: 0x5dabbdcd (stale) Mirror 11: 0x5dabbdcd (stale) Mirror 12: 0x5dabbdcd (stale) Mirror 13: 0x5dabbdcd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000811 1787660055.898995 trunc from 0x040000 to 0x004731 (0x3b8ce bytes) 000812 1787660055.956298 mapwrite 0x022ba6 thru 0x022ed4 (0x32f bytes) 000813 1787660056.085093 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22ed5): Mirror 7: 0xba58ac22 (stale) Mirror 8: 0xba58ac22 (stale) Mirror 10: 0xba58ac22 (stale) Mirror 11: 0xba58ac22 (stale) Mirror 12: 0xba58ac22 (stale) Mirror 13: 0xba58ac22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22ed5): Mirror 7: 0xba58ac22 (stale) Mirror 8: 0xba58ac22 (stale) Mirror 10: 0xba58ac22 (stale) Mirror 11: 0xba58ac22 (stale) Mirror 12: 0xba58ac22 (stale) Mirror 13: 0xba58ac22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22ed5: skipped 000814 1787660056.747200 trunc from 0x022ed5 to 0x03e985 (0x1bab1 bytes) 000817 1787660056.775006 trunc from 0x03e986 to 0x004e4c (0x39b39 bytes) 000818 1787660056.814783 trunc from 0x004e4d to 0x01a205 (0x153b9 bytes) 000821 1787660056.834757 read 0x00f97e thru 0x01a205 (0xa888 bytes) 000823 1787660056.848906 trunc from 0x01a206 to 0x01cee6 (0x2ce1 bytes) 000824 1787660056.872520 trunc from 0x01cee7 to 0x0171a6 (0x5d40 bytes) 000825 1787660056.897398 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x171a7): Mirror 7: 0xf6d0a6d5 (stale) Mirror 8: 0xf6d0a6d5 (stale) Mirror 10: 0xf6d0a6d5 (stale) Mirror 11: 0xf6d0a6d5 (stale) Mirror 12: 0xf6d0a6d5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x171a7): Mirror 7: 0xf6d0a6d5 (stale) Mirror 8: 0xf6d0a6d5 (stale) Mirror 10: 0xf6d0a6d5 (stale) Mirror 11: 0xf6d0a6d5 (stale) Mirror 12: 0xf6d0a6d5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x171a7: skipped 000826 1787660057.489652 mapread 0x016543 thru 0x0171a6 (0xc64 bytes) 000828 1787660057.523555 trunc from 0x0171a7 to 0x037bb9 (0x20a13 bytes) 000830 1787660057.556467 trunc from 0x037bba to 0x03d1b2 (0x55f9 bytes) 000831 1787660057.596708 trunc from 0x03d1b3 to 0x021317 (0x1be9b bytes) 000832 1787660057.613596 read 0x00b422 thru 0x00eb34 (0x3713 bytes) 000833 1787660057.627598 mapwrite 0x02ea2d thru 0x03af10 (0xc4e4 bytes) 000834 1787660057.811997 MIRROR_RESYNC 6 mirrors 000838 1787660060.313163 mapread 0x022dd5 thru 0x024e5c (0x2088 bytes) 000839 1787660060.337391 trunc from 0x03af11 to 0x0200ef (0x1ae21 bytes) 000840 1787660060.417549 read 0x0119ea thru 0x01789f (0x5eb6 bytes) 000841 1787660060.432999 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x200f0): Mirror 8: 0xb3ccfe4e (stale) Mirror 10: 0xb3ccfe4e (stale) Mirror 11: 0xb3ccfe4e (stale) Mirror 12: 0xb3ccfe4e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x200f0): Mirror 8: 0xb3ccfe4e (stale) Mirror 10: 0xb3ccfe4e (stale) Mirror 11: 0xb3ccfe4e (stale) Mirror 12: 0xb3ccfe4e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x200f0: skipped 000843 1787660060.922770 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x200f0): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x200f0): Mirror 8: 0xb3ccfe4e (stale) Mirror 10: 0xb3ccfe4e (stale) Mirror 11: 0xb3ccfe4e (stale) Mirror 12: 0xb3ccfe4e (stale) CRC-32 checksum value for chunk [0, 0x200f0): Mirror 1: 0xb3ccfe4e Mirror 13: 0xb3ccfe4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x200f0: skipped 000844 1787660062.439608 read 0x011b60 thru 0x0200ef (0xe590 bytes) 000847 1787660062.486812 read 0x005b40 thru 0x00c245 (0x6706 bytes) 000849 1787660062.517556 read 0x003064 thru 0x00ff09 (0xcea6 bytes) 000850 1787660062.535600 MIRROR_RESYNC 6 mirrors 000852 1787660063.952096 read 0x01103b thru 0x0176ac (0x6672 bytes) 000853 1787660063.971451 write 0x022dbb thru 0x02ebb4 (0xbdfa bytes) 000854 1787660064.090307 read 0x01676f thru 0x019960 (0x31f2 bytes) 000855 1787660064.114652 MIRROR_RESYNC 6 mirrors 000857 1787660066.121781 MIRROR_RESYNC 6 mirrors 000859 1787660066.699166 write 0x00e292 thru 0x015da0 (0x7b0f bytes) 000860 1787660066.870593 mapread 0x020276 thru 0x02995a (0x96e5 bytes) 000861 1787660066.915879 MIRROR_RESYNC 6 mirrors 000864 1787660069.409558 mapread 0x00355c thru 0x013505 (0xffaa bytes) 000865 1787660069.458191 mapwrite 0x0309e1 thru 0x03b0d4 (0xa6f4 bytes) 000866 1787660069.684273 mapwrite 0x024269 thru 0x028948 (0x46e0 bytes) 000867 1787660069.742711 mapread 0x024aad thru 0x0317ce (0xcd22 bytes) 000869 1787660069.744571 read 0x01823f thru 0x01a148 (0x1f0a bytes) 000870 1787660069.759263 MIRROR_RESYNC 6 mirrors 000872 1787660072.837044 mapread 0x01b835 thru 0x028069 (0xc835 bytes) 000874 1787660072.906302 mapread 0x030e83 thru 0x03b0d4 (0xa252 bytes) 000876 1787660072.929276 mapread 0x03894a thru 0x03a50c (0x1bc3 bytes) 000877 1787660072.929965 write 0x02468a thru 0x029a9d (0x5414 bytes) 000880 1787660073.046050 mapread 0x0331a5 thru 0x03b0d4 (0x7f30 bytes) 000881 1787660073.057945 read 0x01ba8f thru 0x024de5 (0x9357 bytes) 000886 1787660073.070914 mapwrite 0x009c76 thru 0x00eba1 (0x4f2c bytes) 000887 1787660073.148272 mapread 0x037674 thru 0x03b0d4 (0x3a61 bytes) 000888 1787660073.150505 mapread 0x00b59d thru 0x015dd0 (0xa834 bytes) 000889 1787660073.165853 mapwrite 0x01b845 thru 0x0274d1 (0xbc8d bytes) 000890 1787660073.266195 MIRROR_RESYNC 6 mirrors 000892 1787660075.899018 write 0x02adb3 thru 0x035603 (0xa851 bytes) 000894 1787660075.968698 mapread 0x037b15 thru 0x03b0d4 (0x35c0 bytes) 000895 1787660075.988443 write 0x017429 thru 0x01b50e (0x40e6 bytes) 000898 1787660076.000204 mapread 0x025304 thru 0x031f08 (0xcc05 bytes) 000900 1787660076.018062 trunc from 0x03b0d5 to 0x026702 (0x149d2 bytes) 000901 1787660076.049338 write 0x0243ab thru 0x032419 (0xe06f bytes) 000902 1787660076.054996 mapwrite 0x0007c3 thru 0x00423e (0x3a7c bytes) 000903 1787660076.148957 read 0x0064cf thru 0x0065ac (0xde bytes) 000904 1787660076.149898 read 0x01433d thru 0x01ec94 (0xa958 bytes) 000905 1787660076.160394 mapread 0x029517 thru 0x02e68d (0x5177 bytes) 000906 1787660076.163302 mapread 0x0213d1 thru 0x024be3 (0x3813 bytes) 000907 1787660076.165699 read 0x008881 thru 0x012dde (0xa55e bytes) 000909 1787660076.172328 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3241a): Mirror 10: 0xae62b2d7 (stale) Mirror 11: 0xae62b2d7 (stale) Mirror 12: 0xae62b2d7 (stale) Mirror 13: 0xae62b2d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3241a): Mirror 10: 0xae62b2d7 (stale) Mirror 11: 0xae62b2d7 (stale) Mirror 12: 0xae62b2d7 (stale) Mirror 13: 0xae62b2d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3241a: skipped 000910 1787660076.644919 MIRROR_RESYNC 5 mirrors 000911 1787660078.532313 mapwrite 0x03bbbb thru 0x03ffff (0x4445 bytes) 000912 1787660078.821870 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xf3be3d3c (stale) Mirror 12: 0xf3be3d3c (stale) Mirror 13: 0xf3be3d3c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xf3be3d3c (stale) Mirror 12: 0xf3be3d3c (stale) Mirror 13: 0xf3be3d3c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000915 1787660079.245140 write 0x02fbce thru 0x037be1 (0x8014 bytes) 000919 1787660079.263226 trunc from 0x040000 to 0x03d038 (0x2fc7 bytes) 000920 1787660079.286039 trunc from 0x03d039 to 0x0296a6 (0x13992 bytes) 000924 1787660079.311262 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x296a7): Mirror 12: 0xf290cc49 (stale) Mirror 13: 0xf290cc49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x296a7): Mirror 12: 0xf290cc49 (stale) Mirror 13: 0xf290cc49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x296a7: skipped 000925 1787660079.660046 read 0x026f35 thru 0x0296a6 (0x2772 bytes) 000926 1787660079.675153 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x296a7): Mirror 13: 0xf290cc49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x296a7): Mirror 13: 0xf290cc49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x296a7: skipped 000929 1787660080.055510 mapread 0x010396 thru 0x012931 (0x259c bytes) 000930 1787660080.085771 mapwrite 0x010488 thru 0x01466d (0x41e6 bytes) 000932 1787660080.191261 read 0x006893 thru 0x014dc2 (0xe530 bytes) 000934 1787660080.210210 write 0x02f610 thru 0x036570 (0x6f61 bytes) 000941 1787660080.218800 read 0x0092b8 thru 0x012be8 (0x9931 bytes) 000943 1787660080.221952 read 0x00a289 thru 0x00ab96 (0x90e bytes) 000944 1787660080.223285 mapwrite 0x003363 thru 0x004e3a (0x1ad8 bytes) 000945 1787660080.356830 mapwrite 0x00091f thru 0x006bba (0x629c bytes) 000946 1787660080.428527 mapread 0x01c9ea thru 0x023f27 (0x753e bytes) 000948 1787660080.431434 read 0x02805f thru 0x029b2a (0x1acc bytes) 000949 1787660080.432563 mapwrite 0x0036da thru 0x00c3c9 (0x8cf0 bytes) 000950 1787660080.501952 trunc from 0x036571 to 0x035d72 (0x7fe bytes) 000951 1787660080.526203 mapwrite 0x01fb63 thru 0x02ec93 (0xf131 bytes) 000952 1787660080.635583 write 0x029244 thru 0x02b7cf (0x258c bytes) 000953 1787660080.638385 read 0x018f58 thru 0x020430 (0x74d9 bytes) 000954 1787660080.640263 mapwrite 0x000553 thru 0x008fc0 (0x8a6e bytes) 000957 1787660080.723421 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) 000960 1787660080.908999 write 0x01d8f4 thru 0x024246 (0x6953 bytes) 000963 1787660080.926329 trunc from 0x035d73 to 0x01db5e (0x18214 bytes) 000964 1787660080.950011 read 0x009aec thru 0x00b89b (0x1db0 bytes) 000965 1787660080.960487 trunc from 0x01db5f to 0x036b9b (0x1903d bytes) 000966 1787660080.989356 mapread 0x01ccbd thru 0x02bcf0 (0xf034 bytes) 000968 1787660081.043823 trunc from 0x036b9c to 0x021681 (0x1551a bytes) 000969 1787660081.066897 mapread 0x011c9c thru 0x015a90 (0x3df5 bytes) 000971 1787660081.080889 write 0x02ac1f thru 0x02c89a (0x1c7c bytes) 000972 1787660081.082041 mapwrite 0x028c73 thru 0x03579c (0xcb2a bytes) 000977 1787660081.277455 mapwrite 0x032863 thru 0x033d63 (0x1501 bytes) 000981 1787660081.352497 mapwrite 0x03d427 thru 0x03ffff (0x2bd9 bytes) 000982 1787660081.447530 write 0x022d9f thru 0x03027f (0xd4e1 bytes) 000983 1787660081.456539 MIRROR_EXTEND to 2 mirrors 000984 1787660082.646906 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x122:0x0]: Invalid argument (22) 000985 1787660082.870393 mapread 0x0104ce thru 0x01e47f (0xdfb2 bytes) 000987 1787660082.905555 mapwrite 0x01f7c4 thru 0x026e07 (0x7644 bytes) 000988 1787660083.042163 trunc from 0x040000 to 0x008f48 (0x370b7 bytes) 000990 1787660083.071946 trunc from 0x008f49 to 0x007c5c (0x12ec bytes) 000993 1787660083.093883 mapread 0x001643 thru 0x007c5c (0x661a bytes) 000995 1787660083.104938 read 0x000ef1 thru 0x007c5c (0x6d6c bytes) 000996 1787660083.111017 trunc from 0x007c5d to 0x02c196 (0x2453a bytes) 000997 1787660083.132039 write 0x02ea8e thru 0x02f708 (0xc7b bytes) 000999 1787660083.136957 write 0x034b6f thru 0x038df4 (0x4286 bytes) All operations completed A-OK! PASS 70a (154s) == sanity-flr test 71: check mirror extend parent fid ==== 08:14:48 (1787660088) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 08:14:52 (1787660092) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.231133 s, 9.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167472 s, 6.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 08:15:02 (1787660102) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7543680 .. Extending file size to 4467424 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7733696 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8207840 .. Extending file size to 8396128 .. Extending file size to 9237568 .. Extending file size to 8379008 .. Extending file size to 8678400 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8847424 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59473 59474 59477 59478 59479 done 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) 192.168.204.141@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (90s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:16:32 (1787660192) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3463904 .. Extending file size to 7488640 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7362336 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9046112 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8585568 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9145216 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 60907 60908 60909 60911 60914 failed 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) 192.168.204.141@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:17:51 (1787660271) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210560 2432 2206080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210560 2432 2206080 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162676 s, 12.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.354481 s, 5.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xdbb5fec6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:18:13 (1787660293) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122324 s, 17.1 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 08:18:23 (1787660303) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x59:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:18:32 (1787660312) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.31605 s, 6.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x18e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:18:42 (1787660322) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.318552 s, 6.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:18:52 (1787660332) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.219162 s, 9.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:19:28 (1787660368) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126795 s, 8.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:20:31 (1787660431) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.509062 s, 4.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (66s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:21:37 (1787660497) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg441-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg441-server: Pool lustre.test_204f created Adding targets to pool oleg441-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg441-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.456766 s, 4.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg441-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg441-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg441-server: Pool lustre.test_204f destroyed PASS 204f (90s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:23:07 (1787660587) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:23:17 (1787660597) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:23:25 (1787660605) oleg441-server: Pool lustre.test_206 created Adding targets to pool oleg441-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg441-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg441-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg441-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg441-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:23:57 (1787660637) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905258 s, 11.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420579 s, 24.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x95afb610 Mirror 3: 0x95afb610 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:24:10 (1787660650) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:24:14 (1787660654) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:24:19 (1787660659) Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.155203 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x900000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x820000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x820000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x820000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1210000 Read this (4 bytes): BBBB 192.168.204.141@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg441-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:25:02 (1787660702) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0154786 s, 529 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:25:11 (1787660711) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771128 s, 13.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7c lmm_fid: [0x200000405:0x7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7c lmm_fid: [0x200000405:0x7c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611260 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-25 08:25:13.000000000 -0400 Modify: 2026-08-25 08:25:13.000000000 -0400 Change: 2026-08-25 08:25:15.000000000 -0400 Birth: 2026-08-25 08:25:13.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:25:21 (1787660721) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:25:25 (1787660725) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.909018 s, 45.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0721217 s, 56.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 25 08:25 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x7e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 25 08:25 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:25:36 (1787660736) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (11s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:25:47 (1787660747) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 2074 sec =========== 08:26:00 (1787660760) === sanity-flr: start cleanup 08:26:02 (1787660762) === === sanity-flr: finish cleanup 08:26:13 (1787660773) ===