-----============= acceptance-small: sanity-flr ============----- Fri Jun 26 09:32:57 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg258-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:33:21 (1782480801) === oleg258-client.virtnet: executing check_config_client /mnt/lustre oleg258-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg258-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff972dc3b97000.idle_timeout=debug osc.lustre-OST0001-osc-ffff972dc3b97000.idle_timeout=debug disable quota as required oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:34:01 (1782480841) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8253] == sanity-flr test 0a: lfs mirror create with -N option == 09:34:03 (1782480843) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:34:20 (1782480860) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:34:24 (1782480864) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:34:29 (1782480869) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:34:51 (1782480891) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:34:53 (1782480893) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 09:34:56 (1782480896) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff972dc3b97000.stats=clear osc.lustre-OST0001-osc-ffff972dc3b97000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:35:04 (1782480904) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:35:13 (1782480913) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:35:21 (1782480921) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:35:24 (1782480924) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:35:31 (1782480931) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 09:35:39 (1782480939) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:35:45 (1782480945) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:35:53 (1782480953) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:36:00 (1782480960) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:36:09 (1782480969) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:36:15 (1782480975) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.511514 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833703 s, 12.6 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:36:22 (1782480982) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.162918 s, 38.6 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:36:28 (1782480988) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (8s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:36:36 (1782480996) Stopping /mnt/lustre-ost1 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff972dc3b97000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff972dc3b97000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff972dc3b97000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff972dc3b97000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg258-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (129s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:38:45 (1782481125) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.809947 s, 14.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff972dc3b97000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff972dc3b97000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg258-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:39:30 (1782481170) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff972dc3b97000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff972dc3b97000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg258-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff972dc3b97000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff972dc3b97000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg258-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (72s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:40:42 (1782481242) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.28947 s, 22.9 MB/s -rw-r--r-- 1 root root 52428800 Jun 26 09:40 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.827s user 0m0.001s sys 0m0.149s Stopping /mnt/lustre-ost1 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff972dc3b97000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff972dc3b97000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.841s user 0m0.001s sys 0m0.159s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg258-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff972dc3b97000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff972dc3b97000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.150s user 0m0.000s sys 0m0.158s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg258-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg258-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff972dc3b97000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff972dc3b97000.ost_server_uuid in FULL state after 0 sec PASS 33b (83s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:42:05 (1782481325) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:42:08 (1782481328) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:42:11 (1782481331) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 09:42:14 (1782481334) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 09:42:20 (1782481340) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.612665 s, 20.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:42:30 (1782481350) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313746 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490645 s, 21.4 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:42:36 (1782481356) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:42:44 (1782481364) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000692589 s, 1.5 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00188079 s, 544 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000852986 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00215955 s, 474 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 09:42:52 (1782481372) create test files with size 15760177 .. 32561a20a37979f4033f0772d5ff5ad3 - ac27b77df70c0ad42fafd5a19ec90780 - 38a72f6e239fc045f7304010239fbcba - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg258-client.virtnet /mnt/lustre (opts:) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 09:43:13 (1782481393) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:43:25 (1782481405) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:43:30 (1782481410) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745388 s, 14.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022055 s, 47.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 09:43:38 (1782481418) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0480798 s, 43.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.302398 s, 13.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138007 s, 15.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 09:43:51 (1782481431) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 09:43:55 (1782481435) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:43:58 (1782481438) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172315 s, 60.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 09:44:04 (1782481444) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.189557 s, 16.6 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:44:13 (1782481453) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xda:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:44:21 (1782481461) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.417722 s, 25.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg258-client.virtnet /mnt/lustre (opts:) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:44:32 (1782481472) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff972dc5ab0000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.302536 s, 34.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg258-client.virtnet /mnt/lustre (opts:) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff972dc2e79000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:44:45 (1782481485) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (32s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:45:17 (1782481517) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:45:47 (1782481547) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 09:45:51 (1782481551) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.212228 s, 14.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.183342 s, 17.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x23f5dfa0 (nosync,stale) Mirror 2: 0x23f5dfa0 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2c5b029a30cfd5e7fcd1d945f4c40eca - mirror 1 checksum: 3dff05e931f2750380639e7843ec430a - mirror 2 checksum: 2c5b029a30cfd5e7fcd1d945f4c40eca - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-26T09:45:54-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-26T09:45:54-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2c5b029a30cfd5e7fcd1d945f4c40eca - mirror 1 checksum: 3dff05e931f2750380639e7843ec430a - mirror 2 checksum: 3dff05e931f2750380639e7843ec430a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-26T09:45:54-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-26T09:45:54-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 09:46:01 (1782481561) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282254 s, 37.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:46:08 (1782481568) Failing mds1 on oleg258-server Stopping /mnt/lustre-mds1 (opts:) on oleg258-server 09:46:20 (1782481580) shut down facet: mds1 facet_host: oleg258-server facet_failover_host: oleg258-server Failover mds1 to oleg258-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-MDT0000 09:46:51 (1782481611) targets are mounted 09:46:51 (1782481611) facet_failover done oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318003 s, 33.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:47:08 (1782481628) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0350381 s, 585 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0105479 s, 1.9 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:47:18 (1782481638) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0433783 s, 472 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00971262 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:47:27 (1782481647) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.006352 s, 2.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:47:35 (1782481655) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0353246 s, 580 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:47:43 (1782481663) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0176826 s, 232 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0358892 s, 114 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:47:50 (1782481670) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (18s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:48:08 (1782481688) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:48:35 (1782481715) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:48:58 (1782481738) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:49:21 (1782481761) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 09:49:27 (1782481767) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57396 mirror split pid 57397 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57396 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57397 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 09:50:33 (1782481833) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 1835 000001 1782481835.641430 mapwrite 0x00f271 thru 0x01d09c (0xde2c bytes) fallocating to largest ever: 0x40000 000002 1782481835.703324 fallocate 0x035bd0 thru 0x03ffff (0xa430 bytes) 000003 1782481835.707129 trunc from 0x03dc6f to 0x03ffff (0x2391 bytes) 000004 1782481835.711061 mapread 0x0083a3 thru 0x016ba6 (0xe804 bytes) 000005 1782481835.716851 mapwrite 0x039d35 thru 0x03ffff (0x62cb bytes) 000006 1782481835.732457 trunc from 0x040000 to 0x035295 (0xad6a bytes) 000007 1782481835.740585 write 0x0160dc thru 0x016222 (0x147 bytes) 000009 1782481835.742307 mapread 0x00414e thru 0x00a0e8 (0x5f9b bytes) 000010 1782481835.743294 read 0x00748f thru 0x012e17 (0xb989 bytes) 000012 1782481835.744327 fallocate 0x005ef5 thru 0x00c394 (0x64a0 bytes) 000013 1782481835.747461 trunc from 0x028adb to 0x02a56b (0x1a91 bytes) 000014 1782481835.750366 mapwrite 0x005711 thru 0x006710 (0x1000 bytes) 000015 1782481835.763856 fallocate 0x013fc1 thru 0x01de51 (0x9e91 bytes) 000016 1782481835.768128 trunc from 0x035296 to 0x016b16 (0x1e77f bytes) 000017 1782481835.781105 read 0x00cba8 thru 0x013819 (0x6c72 bytes) 000018 1782481835.786448 MIRROR_EXTEND to 2 mirrors 000019 1782481836.304402 read 0x00ad14 thru 0x00ea7f (0x3d6c bytes) 000020 1782481836.311583 mapwrite 0x00cfae thru 0x01ab94 (0xdbe7 bytes) 000021 1782481836.403557 write 0x03ac5d thru 0x03ffff (0x53a3 bytes) 000022 1782481836.405133 trunc from 0x040000 to 0x035211 (0xadee bytes) 000024 1782481836.421452 trunc from 0x035212 to 0x00b351 (0x29ec0 bytes) 000025 1782481836.438650 punch from 0x005056 to 0x00b351 (0x62fc bytes) 000027 1782481836.445653 fallocate 0x01cb17 thru 0x026c25 (0xa10f bytes) 000028 1782481836.449932 punch from 0x00bd80 to 0x00e547 (0x27c8 bytes) 000031 1782481836.460139 mapwrite 0x00de47 thru 0x019cc4 (0xbe7e bytes) 000032 1782481836.491877 trunc from 0x00b27f to 0x00e68e (0x3410 bytes) 000033 1782481836.499729 write 0x01d0ca thru 0x01fa3b (0x2972 bytes) 000034 1782481836.506030 read 0x007f59 thru 0x013c4d (0xbcf5 bytes) 000035 1782481836.540082 mapread 0x01a10c thru 0x01b82e (0x1723 bytes) 000037 1782481836.549270 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26c26): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26c26): Mirror 2: 0x6fea8696 (stale) CRC-32 checksum value for chunk [0, 0x26c26): Mirror 1: 0xa931428c Mirror 3: 0xa931428c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26c26: skipped 000038 1782481837.279609 mapwrite 0x02a78d thru 0x02f5ce (0x4e42 bytes) 000039 1782481837.389542 mapwrite 0x00b26a thru 0x00c2f8 (0x108f bytes) 000040 1782481837.409188 trunc from 0x010074 to 0x0188de (0x886b bytes) 000041 1782481837.418221 read 0x0271b6 thru 0x02f5ce (0x8419 bytes) 000042 1782481837.419221 punch from 0x001326 to 0x00c624 (0xb2ff bytes) 000043 1782481837.424901 fallocate 0x00d7b9 thru 0x013a80 (0x62c8 bytes) 000044 1782481837.430251 mapwrite 0x0333df thru 0x0399ee (0x6610 bytes) 000045 1782481837.471069 mapwrite 0x03f959 thru 0x03ffff (0x6a7 bytes) 000046 1782481837.496640 fallocate 0x0062f6 thru 0x013692 (0xd39d bytes) 000047 1782481837.500539 read 0x034c78 thru 0x03e83e (0x9bc7 bytes) 000048 1782481837.501367 trunc from 0x040000 to 0x03226c (0xdd93 bytes) 000049 1782481837.512785 trunc from 0x03226d to 0x0146a3 (0x1dbc9 bytes) 000050 1782481837.523524 fallocate 0x013733 thru 0x01ead4 (0xb3a2 bytes) 000051 1782481837.530358 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x146a4): Mirror 2: 0x48ff11fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x146a4): Mirror 2: 0x48ff11fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x146a4: skipped 000052 1782481837.732038 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) 000053 1782481837.881276 write 0x0257fd thru 0x02adb1 (0x55b5 bytes) 000054 1782481837.892405 mapwrite 0x009fe8 thru 0x00d351 (0x336a bytes) 000055 1782481837.938084 trunc from 0x014575 to 0x017513 (0x2f9f bytes) 000056 1782481837.944306 punch from 0x0210e3 to 0x0268b5 (0x57d3 bytes) 000057 1782481837.950316 trunc from 0x02adb2 to 0x010463 (0x1a94e bytes) 000058 1782481837.970439 punch from 0x00cb94 to 0x010463 (0x38d0 bytes) 000059 1782481837.975855 trunc from 0x010464 to 0x027a52 (0x175ef bytes) 000060 1782481837.995668 punch from 0x01042e to 0x0168b2 (0x6485 bytes) 000061 1782481838.000873 trunc from 0x01df60 to 0x026990 (0x8a31 bytes) 000062 1782481838.007237 write 0x00d36b thru 0x012d39 (0x59cf bytes) 000063 1782481838.018550 read 0x00d4e2 thru 0x01b8ee (0xe40d bytes) 000065 1782481838.030907 fallocate 0x0209cd thru 0x02b416 (0xaa4a bytes) 000066 1782481838.036507 punch from 0x00ed12 to 0x01378e (0x4a7d bytes) 000067 1782481838.070918 trunc from 0x02b417 to 0x022f79 (0x849d bytes) 000068 1782481838.095462 trunc from 0x022f7a to 0x030429 (0xd4b0 bytes) 000069 1782481838.116504 read 0x0145a9 thru 0x014883 (0x2db bytes) 000070 1782481838.117472 mapread 0x009985 thru 0x00b489 (0x1b05 bytes) 000071 1782481838.119326 trunc from 0x00a728 to 0x014558 (0x9e31 bytes) 000072 1782481838.126771 fallocate 0x0230ec thru 0x02be6d (0x8d82 bytes) 000073 1782481838.132358 punch from 0x01eca3 to 0x02178e (0x2aec bytes) 000075 1782481838.137162 fallocate 0x017cbd thru 0x0268c0 (0xec04 bytes) 000078 1782481838.147480 mapread 0x01fd12 thru 0x02893c (0x8c2b bytes) 000080 1782481838.160018 fallocate 0x01f5c4 thru 0x029d70 (0xa7ad bytes) 000081 1782481838.164191 trunc from 0x03042a to 0x000008 (0x30421 bytes) 000082 1782481838.184334 punch from 00000000 to 0x000008 (0x9 bytes) 000083 1782481838.190378 mapwrite 0x022f73 thru 0x02e425 (0xb4b3 bytes) 000084 1782481838.284064 trunc from 0x022dc7 to 0x025db7 (0x2ff1 bytes) 000085 1782481838.289792 write 0x01d850 thru 0x026b3a (0x92eb bytes) 000086 1782481838.296090 read 0x001843 thru 0x008a75 (0x7233 bytes) 000087 1782481838.308450 fallocate 0x00900e thru 0x011cd4 (0x8cc7 bytes) 000088 1782481838.314889 fallocate 0x032926 thru 0x039f11 (0x75ec bytes) 000089 1782481838.325647 mapread 0x027f4d thru 0x02e425 (0x64d9 bytes) 000090 1782481838.327003 mapwrite 0x00f1fd thru 0x0178eb (0x86ef bytes) 000091 1782481838.363474 MIRROR_EXTEND to 2 mirrors 000092 1782481838.759585 write 0x03f286 thru 0x03ffff (0xd7a bytes) 000093 1782481838.799144 fallocate 0x002ca7 thru 0x0049dc (0x1d36 bytes) 000094 1782481838.803655 mapwrite 0x035165 thru 0x03f2e0 (0xa17c bytes) 000095 1782481838.847246 trunc from 0x040000 to 0x00a82c (0x357d3 bytes) 000096 1782481838.893814 mapwrite 0x034e60 thru 0x03ffff (0xb1a0 bytes) 000097 1782481838.947826 write 0x03f4dc thru 0x03ffff (0xb24 bytes) 000098 1782481838.949454 trunc from 0x040000 to 0x0365fc (0x9a03 bytes) 000099 1782481838.966228 trunc from 0x028fae to 0x0365fc (0xd64f bytes) 000100 1782481838.971411 read 0x023ef0 thru 0x02ea75 (0xab86 bytes) 000101 1782481838.980241 trunc from 0x0365fd to 0x00d10c (0x294f0 bytes) 000102 1782481839.005030 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) 000103 1782481839.119821 write 0x008e37 thru 0x0145b0 (0xb77a bytes) 000104 1782481839.133077 punch from 0x0018e3 to 0x00506c (0x378a bytes) 000105 1782481839.137464 mapwrite 0x00ab72 thru 0x012231 (0x76c0 bytes) 000106 1782481839.155361 write 0x01cbee thru 0x0235f5 (0x6a08 bytes) 000107 1782481839.156871 punch from 0x00b3ca to 0x00b470 (0xa7 bytes) 000109 1782481839.166906 MIRROR_EXTEND to 2 mirrors 000110 1782481839.852309 write 0x03575d thru 0x03ffff (0xa8a3 bytes) 000111 1782481839.885267 mapwrite 0x011708 thru 0x013335 (0x1c2e bytes) 000112 1782481839.926344 fallocate 0x035202 thru 0x03aeaa (0x5ca9 bytes) 000113 1782481839.931085 write 0x017137 thru 0x01e174 (0x703e bytes) 000114 1782481839.936790 punch from 0x00f71d to 0x01dcbe (0xe5a2 bytes) 000115 1782481839.955247 fallocate 0x00dd95 thru 0x016f6a (0x91d6 bytes) 000116 1782481839.966408 write 0x0302b7 thru 0x0361a1 (0x5eeb bytes) 000118 1782481839.982139 mapwrite 0x00d224 thru 0x019d75 (0xcb52 bytes) 000119 1782481840.066106 write 0x02a5e1 thru 0x0309a8 (0x63c8 bytes) 000120 1782481840.075759 fallocate 0x002162 thru 0x00edd6 (0xcc75 bytes) 000121 1782481840.095301 mapwrite 0x0190ad thru 0x028ba1 (0xfaf5 bytes) 000122 1782481840.164961 mapwrite 0x01ce47 thru 0x01d8d4 (0xa8e bytes) 000123 1782481840.193303 punch from 0x00be7b to 0x00d333 (0x14b9 bytes) 000124 1782481840.208383 mapread 0x0225ad thru 0x02db8c (0xb5e0 bytes) 000125 1782481840.230116 fallocate 0x032db3 thru 0x03daa4 (0xacf2 bytes) 000126 1782481840.236261 trunc from 0x03017f to 0x03f398 (0xf21a bytes) 000127 1782481840.242272 mapwrite 0x01ee70 thru 0x02d9e3 (0xeb74 bytes) 000128 1782481840.299373 trunc from 0x040000 to 0x01139c (0x2ec63 bytes) 000129 1782481840.339294 trunc from 0x00741c to 0x00bf06 (0x4aeb bytes) 000130 1782481840.351340 read 0x005c15 thru 0x00b882 (0x5c6e bytes) 000131 1782481840.367243 fallocate 0x01d75a thru 0x024f38 (0x77df bytes) 000132 1782481840.372074 trunc from 0x01139d to 0x01338c (0x1ff0 bytes) 000133 1782481840.387689 mapread 0x00cf13 thru 0x00d174 (0x262 bytes) 000134 1782481840.395006 read 0x011181 thru 0x01338c (0x220c bytes) 000135 1782481840.395813 punch from 0x001073 to 0x00de1f (0xcdad bytes) 000136 1782481840.401066 trunc from 0x00883d to 0x0105ee (0x7db2 bytes) 000137 1782481840.405264 mapread 0x008c60 thru 0x01338c (0xa72d bytes) 000138 1782481840.419887 mapwrite 0x034d09 thru 0x03ffff (0xb2f7 bytes) 000140 1782481840.501225 read 0x01c6eb thru 0x027883 (0xb199 bytes) 000141 1782481840.502070 trunc from 0x040000 to 0x00ab77 (0x35488 bytes) 000143 1782481840.513959 read 0x00227a thru 0x00ab77 (0x88fe bytes) 000144 1782481840.514463 mapread 0x009861 thru 0x00ab77 (0x1317 bytes) 000145 1782481840.515445 mapread 0x00748a thru 0x00ab77 (0x36ee bytes) 000146 1782481840.516362 trunc from 0x00ab78 to 0x02ae97 (0x20320 bytes) 000147 1782481840.528677 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ae98): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ae98): Mirror 2: 0x309e24c2 (stale) CRC-32 checksum value for chunk [0, 0x2ae98): Mirror 1: 0xb94b7ec0 Mirror 3: 0xb94b7ec0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ae98: skipped 000148 1782481841.229906 write 0x033c56 thru 0x03ffff (0xc3aa bytes) 000149 1782481841.263060 write 0x01900b thru 0x019209 (0x1ff bytes) 000150 1782481841.268605 fallocate 0x026445 thru 0x028711 (0x22cd bytes) 000151 1782481841.276090 MIRROR_RESYNC 3 mirrors 000152 1782481841.937096 mapwrite 0x02fff8 thru 0x0312da (0x12e3 bytes) 000153 1782481842.009775 mapwrite 0x0184c2 thru 0x01bd63 (0x38a2 bytes) 000155 1782481842.041342 write 0x02f7b6 thru 0x03609f (0x68ea bytes) 000156 1782481842.042833 punch from 0x0023e6 to 0x00c2d6 (0x9ef1 bytes) 000157 1782481842.048999 trunc from 0x02f624 to 0x030196 (0xb73 bytes) 000158 1782481842.066584 MIRROR_RESYNC 3 mirrors 000159 1782481842.674765 read 0x01e6ad thru 0x021dbd (0x3711 bytes) 000160 1782481842.682372 fallocate 0x019316 thru 0x01ae25 (0x1b10 bytes) 000161 1782481842.720208 read 0x023993 thru 0x026fb7 (0x3625 bytes) 000162 1782481842.736971 read 0x03ef39 thru 0x03ffff (0x10c7 bytes) 000163 1782481842.737947 mapwrite 0x018530 thru 0x018b93 (0x664 bytes) 000164 1782481842.756248 write 0x02c3f8 thru 0x034526 (0x812f bytes) 000165 1782481842.758742 mapread 0x032750 thru 0x03a5af (0x7e60 bytes) 000166 1782481842.762425 punch from 0x02bec3 to 0x02f7b0 (0x38ee bytes) 000167 1782481842.782667 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6bc6b5e (stale) Mirror 3: 0x6bc6b5e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7fd160a8 Mirror 4: 0x7fd160a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000168 1782481843.446976 fallocate 0x018daa thru 0x0210c5 (0x831c bytes) 000169 1782481843.485959 read 0x002141 thru 0x00cb74 (0xaa34 bytes) 000170 1782481843.493586 mapread 0x0070c5 thru 0x009b14 (0x2a50 bytes) 000171 1782481843.495505 mapwrite 0x014b98 thru 0x0182bc (0x3725 bytes) 000172 1782481843.527324 fallocate 0x03677e thru 0x03e559 (0x7ddc bytes) 000173 1782481843.535279 write 0x02318c thru 0x025e9b (0x2d10 bytes) 000174 1782481843.547315 fallocate 0x031e71 thru 0x0378cc (0x5a5c bytes) 000175 1782481843.551466 mapread 0x023cd7 thru 0x02fd29 (0xc053 bytes) 000176 1782481843.567300 write 0x0366c2 thru 0x03ffff (0x993e bytes) 000177 1782481843.570202 trunc from 0x040000 to 0x023f49 (0x1c0b6 bytes) 000178 1782481843.586690 trunc from 0x023f4a to 0x0367b0 (0x12867 bytes) 000179 1782481843.601491 write 0x033955 thru 0x03579a (0x1e46 bytes) 000180 1782481843.610206 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x367b1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367b1): Mirror 2: 0xc1642faa (stale) Mirror 3: 0xc1642faa (stale) Mirror 4: 0x99a51f4e (stale) CRC-32 checksum value for chunk [0, 0x367b1): Mirror 1: 0xf5e9b47e Mirror 5: 0xf5e9b47e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367b1: skipped 000181 1782481844.277582 mapread 0x02b4e0 thru 0x034dbd (0x98de bytes) 000182 1782481844.309628 mapread 0x023e5d thru 0x02dc7f (0x9e23 bytes) 000183 1782481844.330158 trunc from 0x0367b1 to 0x01fb2a (0x16c86 bytes) 000184 1782481844.396862 punch from 0x01e071 to 0x01fb2a (0x1aba bytes) 000185 1782481844.401056 punch from 0x00bae3 to 0x01614f (0xa66d bytes) 000186 1782481844.405761 mapread 0x00edf4 thru 0x012640 (0x384d bytes) 000187 1782481844.417272 fallocate 0x02db6b thru 0x03c32d (0xe7c3 bytes) 000188 1782481844.421500 read 0x0070d6 thru 0x015f21 (0xee4c bytes) 000189 1782481844.425675 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c32e): Mirror 2: 0xaf4bbfb8 (stale) Mirror 4: 0x1a6c7507 (stale) Mirror 5: 0x28bb7d98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c32e): Mirror 2: 0xaf4bbfb8 (stale) Mirror 4: 0x1a6c7507 (stale) Mirror 5: 0x28bb7d98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c32e: skipped 000190 1782481844.705991 mapwrite 0x02d621 thru 0x037c4f (0xa62f bytes) 000191 1782481844.737977 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c32e): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c32e): Mirror 2: 0xaf4bbfb8 (stale) Mirror 4: 0x1a6c7507 (stale) Mirror 5: 0x28bb7d98 (stale) CRC-32 checksum value for chunk [0, 0x3c32e): Mirror 1: 0x947f0ad6 Mirror 6: 0x947f0ad6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c32e: skipped 000192 1782481845.764470 read 0x02b035 thru 0x031c4d (0x6c19 bytes) 000193 1782481845.784009 punch from 0x008b71 to 0x012092 (0x9522 bytes) 000194 1782481845.835165 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c32e): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c32e): Mirror 2: 0xaf4bbfb8 (stale) Mirror 4: 0x1a6c7507 (stale) Mirror 5: 0x28bb7d98 (stale) Mirror 6: 0x947f0ad6 (stale) CRC-32 checksum value for chunk [0, 0x3c32e): Mirror 1: 0x947f0ad6 Mirror 7: 0x947f0ad6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c32e: skipped 000196 1782481846.673456 write 0x01b775 thru 0x023e42 (0x86ce bytes) 000197 1782481846.722427 read 0x02988e thru 0x032ca2 (0x9415 bytes) 000198 1782481846.736155 write 0x01bf06 thru 0x01c275 (0x370 bytes) 000199 1782481846.737570 fallocate 0x039624 thru 0x03eace (0x54ab bytes) 000200 1782481846.743258 trunc from 0x03c32e to 0x006055 (0x362d8 bytes) 000201 1782481846.773205 trunc from 0x006056 to 0x02a4ff (0x244aa bytes) 000202 1782481846.790129 punch from 0x01de40 to 0x021d30 (0x3ef1 bytes) 000203 1782481846.795472 mapread 0x026126 thru 0x02909e (0x2f79 bytes) 000204 1782481846.817852 MIRROR_RESYNC 6 mirrors 000205 1782481847.692273 trunc from 0x00910f to 0x015d73 (0xcc65 bytes) 000207 1782481847.741789 punch from 0x016929 to 0x01b469 (0x4b41 bytes) 000209 1782481847.746960 trunc from 0x02a500 to 0x02c466 (0x1f67 bytes) 000210 1782481847.769868 punch from 0x005a10 to 0x00c49d (0x6a8e bytes) 000211 1782481847.775239 mapwrite 0x026019 thru 0x035fd3 (0xffbb bytes) 000213 1782481847.848879 read 0x00ab6e thru 0x00f9da (0x4e6d bytes) 000214 1782481847.866685 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x35fd4): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35fd4): Mirror 2: 0x64b27a50 (stale) Mirror 4: 0x64b27a50 (stale) Mirror 5: 0x64b27a50 (stale) Mirror 6: 0x64b27a50 (stale) Mirror 7: 0x64b27a50 (stale) CRC-32 checksum value for chunk [0, 0x35fd4): Mirror 1: 0x2e8995f4 Mirror 8: 0x2e8995f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35fd4: skipped 000215 1782481848.691461 write 0x00f8b8 thru 0x0189a2 (0x90eb bytes) 000216 1782481848.732022 mapread 0x009e90 thru 0x0106c8 (0x6839 bytes) 000217 1782481848.754188 fallocate 0x010b61 thru 0x010e1c (0x2bc bytes) 000218 1782481848.758796 trunc from 0x018685 to 0x021a15 (0x9391 bytes) 000219 1782481848.773969 write 0x029457 thru 0x03859c (0xf146 bytes) 000220 1782481848.779903 fallocate 0x023a14 thru 0x02bafc (0x80e9 bytes) 000221 1782481848.791787 fallocate 0x008a55 thru 0x01807c (0xf628 bytes) 000222 1782481848.798648 punch from 0x00e666 to 0x014221 (0x5bbc bytes) 000223 1782481848.807794 read 0x0225e0 thru 0x024ed4 (0x28f5 bytes) 000225 1782481848.815136 read 0x02078d thru 0x02600c (0x5880 bytes) 000226 1782481848.835976 punch from 0x037a98 to 0x03859c (0xb05 bytes) 000227 1782481848.854341 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3859d): Mirror 2: 0x63231351 (stale) Mirror 4: 0x63231351 (stale) Mirror 6: 0x63231351 (stale) Mirror 7: 0x63231351 (stale) Mirror 8: 0xd96e5654 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3859d): Mirror 2: 0x63231351 (stale) Mirror 4: 0x63231351 (stale) Mirror 6: 0x63231351 (stale) Mirror 7: 0x63231351 (stale) Mirror 8: 0xd96e5654 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3859d: skipped 000228 1782481849.399546 punch from 0x00a2cb to 0x012f09 (0x8c3f bytes) 000229 1782481849.410682 mapwrite 0x03613f thru 0x03ffff (0x9ec1 bytes) 000230 1782481849.514197 write 0x01e32c thru 0x024f21 (0x6bf6 bytes) 000232 1782481849.525788 trunc from 0x040000 to 0x0228a0 (0x1d75f bytes) 000233 1782481849.545431 trunc from 0x02175e to 0x0228a0 (0x1143 bytes) 000234 1782481849.564147 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x228a1): Mirror 2: 0x36eb9f7b (stale) Mirror 6: 0x36eb9f7b (stale) Mirror 7: 0x36eb9f7b (stale) Mirror 8: 0x36eb9f7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x228a1): Mirror 2: 0x36eb9f7b (stale) Mirror 6: 0x36eb9f7b (stale) Mirror 7: 0x36eb9f7b (stale) Mirror 8: 0x36eb9f7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x228a1: skipped 000235 1782481849.962596 read 0x006e71 thru 0x015e2a (0xefba bytes) 000236 1782481849.975020 read 0x013cdd thru 0x013ed4 (0x1f8 bytes) 000237 1782481849.975966 trunc from 0x012ea3 to 0x019bd8 (0x6d36 bytes) 000238 1782481849.982916 trunc from 0x00d309 to 0x017e6b (0xab63 bytes) 000239 1782481849.991268 write 0x030be8 thru 0x03ea90 (0xdea9 bytes) 000240 1782481849.998759 write 0x03d7fb thru 0x03e1a4 (0x9aa bytes) 000242 1782481850.001591 trunc from 0x002a6e to 0x00b356 (0x88e9 bytes) 000243 1782481850.008295 mapread 0x0111eb thru 0x014509 (0x331f bytes) 000244 1782481850.019548 read 0x00c532 thru 0x01781f (0xb2ee bytes) 000245 1782481850.021114 fallocate 0x019fc0 thru 0x022b1b (0x8b5c bytes) 000246 1782481850.026326 trunc from 0x024247 to 0x025e80 (0x1c3a bytes) 000247 1782481850.031813 mapread 0x00d077 thru 0x01cddb (0xfd65 bytes) 000249 1782481850.035654 mapwrite 0x0186f9 thru 0x02640b (0xdd13 bytes) 000250 1782481850.070992 trunc from 0x03ea91 to 0x01e6d3 (0x203bd bytes) 000251 1782481850.098763 mapwrite 0x0273a9 thru 0x036306 (0xef5e bytes) 000252 1782481850.240670 write 0x01b924 thru 0x01e03f (0x271c bytes) 000253 1782481850.242142 trunc from 0x036307 to 0x00c4bd (0x29e49 bytes) 000254 1782481850.260453 punch from 0x005db1 to 0x00a19a (0x43ea bytes) 000255 1782481850.267700 fallocate 0x02e260 thru 0x03082a (0x25cb bytes) 000256 1782481850.275122 trunc from 0x012e68 to 0x01a7b8 (0x7951 bytes) 000257 1782481850.280955 trunc from 0x03082b to 0x03f257 (0xea2d bytes) 000258 1782481850.296251 write 0x02cecc thru 0x034874 (0x79a9 bytes) 000259 1782481850.309660 trunc from 0x03f258 to 0x002675 (0x3cbe2 bytes) 000260 1782481850.335559 MIRROR_RESYNC 5 mirrors 000261 1782481851.057980 fallocate 0x033384 thru 0x037303 (0x3f80 bytes) 000262 1782481851.089217 write 0x030856 thru 0x03fe99 (0xf644 bytes) 000263 1782481851.096622 fallocate 0x03aff0 thru 0x03ffff (0x5010 bytes) 000264 1782481851.101357 mapread 0x01d417 thru 0x01e696 (0x1280 bytes) 000265 1782481851.110568 write 0x022fea thru 0x027ed1 (0x4ee8 bytes) 000266 1782481851.119582 trunc from 0x033e06 to 0x03fe99 (0xc094 bytes) 000267 1782481851.133651 punch from 0x0046f4 to 0x006c24 (0x2531 bytes) 000268 1782481851.137633 fallocate 0x009a7c thru 0x00f6c5 (0x5c4a bytes) 000269 1782481851.142387 trunc from 0x03fe9a to 0x0245fd (0x1b89c bytes) 000270 1782481851.163692 punch from 0x00627d to 0x00ee5c (0x8be0 bytes) 000271 1782481851.168568 mapread 0x0105a6 thru 0x01f482 (0xeedd bytes) 000272 1782481851.193213 read 0x00d9e4 thru 0x0164f0 (0x8b0d bytes) 000274 1782481851.194988 trunc from 0x0159ee to 0x01db71 (0x8184 bytes) 000275 1782481851.200853 punch from 0x0103ac to 0x015e96 (0x5aeb bytes) 000277 1782481851.205398 mapwrite 0x012f55 thru 0x0130c9 (0x175 bytes) 000278 1782481851.222072 write 0x028cb3 thru 0x02c102 (0x3450 bytes) 000279 1782481851.223691 read 0x008adb thru 0x0138da (0xae00 bytes) 000280 1782481851.239426 write 0x007417 thru 0x011fb9 (0xaba3 bytes) 000281 1782481851.245549 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c103): Mirror 6: 0x333dd045 (stale) Mirror 7: 0x333dd045 (stale) Mirror 8: 0x333dd045 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c103): Mirror 6: 0x333dd045 (stale) Mirror 7: 0x333dd045 (stale) Mirror 8: 0x333dd045 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c103: skipped 000283 1782481851.600250 read 0x014202 thru 0x01f0a2 (0xaea1 bytes) 000284 1782481851.612469 mapread 0x020d08 thru 0x026d8e (0x6087 bytes) 000285 1782481851.632304 punch from 0x027d41 to 0x02c102 (0x43c2 bytes) 000286 1782481851.641318 read 0x027e8a thru 0x028375 (0x4ec bytes) 000287 1782481851.652998 write 0x03942c thru 0x03ffff (0x6bd4 bytes) 000289 1782481851.660219 trunc from 0x040000 to 0x03a863 (0x579c bytes) 000290 1782481851.678804 mapwrite 0x02f766 thru 0x0324f5 (0x2d90 bytes) 000291 1782481851.702094 mapwrite 0x025c50 thru 0x03462b (0xe9dc bytes) 000292 1782481851.724530 trunc from 0x0237e1 to 0x0244d9 (0xcf9 bytes) 000293 1782481851.729488 read 0x006be5 thru 0x0087d3 (0x1bef bytes) 000294 1782481851.730239 trunc from 0x00d2e2 to 0x018a1c (0xb73b bytes) 000295 1782481851.741647 trunc from 0x02f3fb to 0x03a863 (0xb469 bytes) 000296 1782481851.751474 read 0x033c07 thru 0x03a863 (0x6c5d bytes) 000298 1782481851.764565 trunc from 0x0304b7 to 0x0394c8 (0x9012 bytes) 000299 1782481851.773584 mapwrite 0x009690 thru 0x00a600 (0xf71 bytes) 000300 1782481851.805297 fallocate 0x023528 thru 0x02d053 (0x9b2c bytes) 000301 1782481851.809972 mapwrite 0x007c5c thru 0x00ac62 (0x3007 bytes) 000302 1782481851.833261 trunc from 0x03a864 to 0x0232be (0x175a5 bytes) 000303 1782481851.857218 mapread 0x00e047 thru 0x0191f5 (0xb1af bytes) 000304 1782481851.858676 trunc from 0x01628d to 0x02056a (0xa2de bytes) 000305 1782481851.878115 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x232bf): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x232bf): Mirror 6: 0xb860a74 (stale) Mirror 7: 0xb860a74 (stale) Mirror 8: 0xb860a74 (stale) CRC-32 checksum value for chunk [0, 0x232bf): Mirror 1: 0x8775e68c Mirror 9: 0x8775e68c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x232bf: skipped 000306 1782481852.816621 mapread 0x0145a8 thru 0x019d4c (0x57a5 bytes) 000307 1782481852.853662 trunc from 0x0232bf to 0x01a1f5 (0x90c9 bytes) 000308 1782481852.897411 trunc from 0x01a1f6 to 0x005f0c (0x142e9 bytes) 000309 1782481852.917245 MIRROR_RESYNC 5 mirrors 000310 1782481853.669375 mapwrite 0x0083a1 thru 0x00f203 (0x6e63 bytes) 000311 1782481853.748940 trunc from 0x00eb33 to 0x00f203 (0x6d1 bytes) 000312 1782481853.754323 write 0x02ce86 thru 0x02e41d (0x1598 bytes) 000313 1782481853.755443 punch from 0x002eef to 0x00c50d (0x961f bytes) 000314 1782481853.764142 write 0x014327 thru 0x022e22 (0xeafc bytes) 000316 1782481853.772535 fallocate 0x031594 thru 0x03ffff (0xea6c bytes) 000317 1782481853.777590 punch from 0x003d9c to 0x008f46 (0x51ab bytes) 000318 1782481853.781787 fallocate 0x028cb3 thru 0x02dc8a (0x4fd8 bytes) 000320 1782481853.787594 read 0x0284ea thru 0x02d76f (0x5286 bytes) 000321 1782481853.793920 fallocate 0x024c2d thru 0x026962 (0x1d36 bytes) 000322 1782481853.798593 mapread 0x015de7 thru 0x01c9ba (0x6bd4 bytes) 000323 1782481853.800412 mapread 0x002b7d thru 0x008b1c (0x5fa0 bytes) 000324 1782481853.809876 fallocate 0x025e75 thru 0x02841e (0x25aa bytes) 000325 1782481853.814412 trunc from 0x02e41e to 0x018aef (0x1592e bytes) 000327 1782481853.833260 write 0x03c12a thru 0x03ffff (0x3ed6 bytes) 000328 1782481853.835297 trunc from 0x040000 to 0x01c141 (0x23ebe bytes) 000329 1782481853.847174 trunc from 0x014241 to 0x01c141 (0x7f01 bytes) 000330 1782481853.856454 write 0x011264 thru 0x01e1f5 (0xcf92 bytes) 000331 1782481853.860408 punch from 0x0021a2 to 0x00c4f1 (0xa350 bytes) 000332 1782481853.865198 write 0x024f31 thru 0x02ca45 (0x7b15 bytes) 000333 1782481853.866516 trunc from 0x011fc3 to 0x01bd7f (0x9dbd bytes) 000334 1782481853.877971 punch from 0x0235c3 to 0x02ca45 (0x9483 bytes) 000335 1782481853.883220 fallocate 0x009c8a thru 0x00fb5a (0x5ed1 bytes) 000336 1782481853.888320 trunc from 0x028468 to 0x02b328 (0x2ec1 bytes) 000338 1782481853.892638 punch from 0x012d61 to 0x017193 (0x4433 bytes) 000339 1782481853.898902 MIRROR_RESYNC 5 mirrors 000340 1782481855.203943 fallocate 0x023c98 thru 0x0293b8 (0x5721 bytes) 000341 1782481855.281016 trunc from 0x02ac02 to 0x02ca45 (0x1e44 bytes) 000342 1782481855.293926 trunc from 0x011ab6 to 0x01af71 (0x94bc bytes) 000343 1782481855.301278 mapread 0x003e26 thru 0x0090eb (0x52c6 bytes) 000344 1782481855.327547 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ca46): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca46): Mirror 6: 0xc423e2bf (stale) Mirror 7: 0xc423e2bf (stale) Mirror 8: 0xc423e2bf (stale) Mirror 9: 0xc423e2bf (stale) CRC-32 checksum value for chunk [0, 0x2ca46): Mirror 1: 0x15f591a6 Mirror 10: 0x15f591a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca46: skipped 000345 1782481856.173862 write 0x0068b1 thru 0x00ca9d (0x61ed bytes) 000346 1782481856.238553 mapread 0x0149fb thru 0x014df3 (0x3f9 bytes) 000347 1782481856.246975 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ca46): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ca46): Mirror 6: 0xc423e2bf (stale) Mirror 7: 0xc423e2bf (stale) Mirror 8: 0xc423e2bf (stale) Mirror 9: 0xc423e2bf (stale) Mirror 10: 0x15f591a6 (stale) CRC-32 checksum value for chunk [0, 0x2ca46): Mirror 1: 0xdebd6010 Mirror 11: 0xdebd6010 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ca46: skipped 000348 1782481857.302268 mapwrite 0x01b3ca thru 0x01f1e7 (0x3e1e bytes) 000349 1782481857.379166 punch from 0x02902e to 0x02967c (0x64f bytes) 000350 1782481857.383489 mapread 0x025915 thru 0x02ca45 (0x7131 bytes) 000351 1782481857.394779 mapwrite 0x036ec1 thru 0x039726 (0x2866 bytes) 000352 1782481857.458923 trunc from 0x012c9a to 0x019980 (0x6ce7 bytes) 000353 1782481857.464130 trunc from 0x03001d to 0x030c28 (0xc0c bytes) 000354 1782481857.472519 MIRROR_RESYNC 7 mirrors 000356 1782481859.902077 punch from 0x00e6ff to 0x01299e (0x42a0 bytes) 000357 1782481859.965514 fallocate 0x01d84b thru 0x029c2c (0xc3e2 bytes) 000358 1782481859.979843 read 0x038640 thru 0x039726 (0x10e7 bytes) 000360 1782481859.987454 read 0x022c55 thru 0x031693 (0xea3f bytes) 000361 1782481860.003535 write 0x01886e thru 0x020883 (0x8016 bytes) 000362 1782481860.015699 MIRROR_RESYNC 7 mirrors 000363 1782481861.977351 mapread 0x005a76 thru 0x012834 (0xcdbf bytes) 000364 1782481862.009618 mapwrite 0x017ce4 thru 0x01931b (0x1638 bytes) 000365 1782481862.080041 read 0x012170 thru 0x01ab27 (0x89b8 bytes) 000366 1782481862.080616 mapread 0x003057 thru 0x00fe0e (0xcdb8 bytes) 000367 1782481862.088340 trunc from 0x039727 to 0x039441 (0x2e5 bytes) 000368 1782481862.102684 punch from 0x001325 to 0x0109a7 (0xf683 bytes) 000369 1782481862.106862 trunc from 0x039442 to 0x01776c (0x21cd5 bytes) 000370 1782481862.116380 trunc from 0x01776d to 0x039149 (0x219dd bytes) 000371 1782481862.131569 mapwrite 0x03dd57 thru 0x03ffff (0x22a9 bytes) 000372 1782481862.159077 trunc from 0x0307d8 to 0x032b3f (0x2368 bytes) 000373 1782481862.162767 trunc from 0x040000 to 0x006bfb (0x39404 bytes) 000374 1782481862.180688 mapwrite 0x02eb05 thru 0x02f1b7 (0x6b3 bytes) 000376 1782481862.222693 mapwrite 0x02799e thru 0x030390 (0x89f3 bytes) 000377 1782481862.269483 punch from 0x00b0e2 to 0x010fb9 (0x5ed8 bytes) 000378 1782481862.274290 mapread 0x02f32c thru 0x030390 (0x1065 bytes) 000379 1782481862.275570 punch from 0x015b42 to 0x018537 (0x29f6 bytes) 000380 1782481862.280221 fallocate 0x0296c9 thru 0x038c5a (0xf592 bytes) 000381 1782481862.287063 trunc from 0x030391 to 0x01483f (0x1bb51 bytes) 000383 1782481862.302070 fallocate 0x003faf thru 0x010823 (0xc875 bytes) 000384 1782481862.310463 MIRROR_RESYNC 7 mirrors 000385 1782481863.114686 punch from 0x013272 to 0x01483f (0x15ce bytes) 000386 1782481863.172914 read 0x007c56 thru 0x012635 (0xa9e0 bytes) 000387 1782481863.182216 write 0x009cea thru 0x016179 (0xc490 bytes) 000388 1782481863.183891 mapread 0x00b49b thru 0x01494b (0x94b1 bytes) 000389 1782481863.186407 trunc from 0x00ae1e to 0x016179 (0xb35c bytes) 000390 1782481863.208526 fallocate 0x03c73d thru 0x03ce3e (0x702 bytes) 000392 1782481863.220338 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ce3f): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce3f): Mirror 6: 0xd3efb253 (stale) Mirror 7: 0xd3efb253 (stale) Mirror 8: 0xd3efb253 (stale) Mirror 9: 0xd3efb253 (stale) Mirror 10: 0xd3efb253 (stale) Mirror 11: 0xd3efb253 (stale) CRC-32 checksum value for chunk [0, 0x3ce3f): Mirror 1: 0x1e96a157 Mirror 12: 0x1e96a157 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce3f: skipped 000393 1782481864.022163 read 0x004ae0 thru 0x01166d (0xcb8e bytes) 000395 1782481864.045229 punch from 0x0306d6 to 0x032a28 (0x2353 bytes) 000397 1782481864.088059 read 0x01e0e4 thru 0x020426 (0x2343 bytes) 000399 1782481864.098562 fallocate 0x03b2ed thru 0x03ffff (0x4d13 bytes) 000400 1782481864.103835 trunc from 0x009379 to 0x01343b (0xa0c3 bytes) 000401 1782481864.109455 fallocate 0x032944 thru 0x03ffff (0xd6bc bytes) 000402 1782481864.113879 fallocate 0x0322a1 thru 0x03d3cd (0xb12d bytes) 000403 1782481864.117857 mapread 0x02d67f thru 0x036769 (0x90eb bytes) 000404 1782481864.119470 fallocate 0x013bad thru 0x0179c4 (0x3e18 bytes) 000405 1782481864.124503 mapwrite 0x025eeb thru 0x0301a9 (0xa2bf bytes) 000406 1782481864.145512 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ce3f): Mirror 6: 0xd3efb253 (stale) Mirror 7: 0xd3efb253 (stale) Mirror 8: 0xd3efb253 (stale) Mirror 10: 0xd3efb253 (stale) Mirror 11: 0xd3efb253 (stale) Mirror 12: 0x1e96a157 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce3f): Mirror 6: 0xd3efb253 (stale) Mirror 7: 0xd3efb253 (stale) Mirror 8: 0xd3efb253 (stale) Mirror 10: 0xd3efb253 (stale) Mirror 11: 0xd3efb253 (stale) Mirror 12: 0x1e96a157 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce3f: skipped 000407 1782481864.770555 mapread 0x029e26 thru 0x0351a8 (0xb383 bytes) 000409 1782481864.801418 trunc from 0x03ce3f to 0x03896d (0x44d1 bytes) 000410 1782481864.842915 write 0x034531 thru 0x035449 (0xf19 bytes) 000411 1782481864.845517 read 0x02d85f thru 0x030688 (0x2e2a bytes) 000412 1782481864.846245 read 0x00e4c8 thru 0x01a633 (0xc16c bytes) 000413 1782481864.861890 trunc from 0x0034c9 to 0x00b872 (0x83aa bytes) 000415 1782481864.871291 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3896e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3896e): Mirror 6: 0x81d46a7a (stale) Mirror 7: 0x81d46a7a (stale) Mirror 8: 0x81d46a7a (stale) Mirror 10: 0x81d46a7a (stale) Mirror 11: 0x81d46a7a (stale) Mirror 12: 0xbcf1645b (stale) CRC-32 checksum value for chunk [0, 0x3896e): Mirror 1: 0xec0eb0e Mirror 13: 0xec0eb0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3896e: skipped 000417 1782481865.790501 mapwrite 0x002dc2 thru 0x00c830 (0x9a6f bytes) 000419 1782481865.930313 read 0x005ccd thru 0x00708f (0x13c3 bytes) 000420 1782481865.931225 mapread 0x009ea3 thru 0x015a07 (0xbb65 bytes) 000421 1782481865.943193 mapread 0x0277fd thru 0x02a662 (0x2e66 bytes) 000422 1782481865.957127 mapread 0x031ce6 thru 0x0381a9 (0x64c4 bytes) 000423 1782481865.969178 mapwrite 0x01468c thru 0x01e7a3 (0xa118 bytes) 000424 1782481865.994013 trunc from 0x03896e to 0x01dd3e (0x1ac2f bytes) 000425 1782481866.014818 mapread 0x00c76d thru 0x00d301 (0xb95 bytes) 000426 1782481866.015953 mapread 0x009bfa thru 0x01743a (0xd841 bytes) 000427 1782481866.017795 punch from 0x009879 to 0x013f3b (0xa6c3 bytes) 000428 1782481866.021898 punch from 0x01c3da to 0x01dd3e (0x1965 bytes) 000429 1782481866.026895 fallocate 0x012a88 thru 0x017465 (0x49de bytes) 000430 1782481866.031082 read 0x015865 thru 0x01dd3e (0x84da bytes) 000431 1782481866.041068 MIRROR_RESYNC 8 mirrors 000432 1782481867.637085 trunc from 0x009c3b to 0x0136c2 (0x9a88 bytes) 000433 1782481867.669211 punch from 0x018352 to 0x01dd3e (0x59ed bytes) 000434 1782481867.675506 punch from 0x01ad3f to 0x01dd3e (0x3000 bytes) 000435 1782481867.680441 mapread 0x003d9f thru 0x0086d6 (0x4938 bytes) 000436 1782481867.700973 trunc from 0x00d6ca to 0x014dd1 (0x7708 bytes) 000437 1782481867.707415 punch from 0x01b108 to 0x01dd3e (0x2c37 bytes) 000438 1782481867.711966 punch from 0x010219 to 0x015a61 (0x5849 bytes) 000439 1782481867.720790 trunc from 0x01dd3f to 0x00a7e9 (0x13555 bytes) 000440 1782481867.738127 write 0x024a50 thru 0x0298a8 (0x4e59 bytes) 000441 1782481867.739524 fallocate 0x01ff21 thru 0x0249ec (0x4acc bytes) 000442 1782481867.744801 punch from 0x00af1a to 0x016836 (0xb91d bytes) 000443 1782481867.749966 trunc from 0x0298a9 to 0x0332bd (0x9a15 bytes) 000444 1782481867.764840 punch from 0x02adda to 0x02fb37 (0x4d5e bytes) 000445 1782481867.769565 mapwrite 0x00b3c8 thru 0x014530 (0x9169 bytes) 000446 1782481867.809331 punch from 0x0007c9 to 0x002a37 (0x226f bytes) 000447 1782481867.814300 fallocate 0x003d71 thru 0x00467b (0x90b bytes) 000448 1782481867.817252 fallocate 0x03cefe thru 0x03fb1e (0x2c21 bytes) 000449 1782481867.820965 mapread 0x029181 thru 0x036ad4 (0xd954 bytes) 000450 1782481867.838312 punch from 0x02b766 to 0x035cb4 (0xa54f bytes) 000451 1782481867.842552 punch from 0x01836c to 0x021be0 (0x9875 bytes) 000452 1782481867.846659 mapread 0x03d7e2 thru 0x03fb1e (0x233d bytes) 000453 1782481867.853296 trunc from 0x03fb1f to 0x022c46 (0x1ced8 bytes) 000454 1782481867.867436 trunc from 0x00c3ba to 0x017edc (0xbb23 bytes) 000456 1782481867.872017 read 0x003af8 thru 0x013160 (0xf669 bytes) 000457 1782481867.880105 write 0x03a76f thru 0x03ffff (0x5891 bytes) 000458 1782481867.885837 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5a10e03e (stale) Mirror 8: 0x5a10e03e (stale) Mirror 10: 0x5a10e03e (stale) Mirror 11: 0x5a10e03e (stale) Mirror 12: 0x5a10e03e (stale) Mirror 13: 0x5a10e03e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x5a10e03e (stale) Mirror 8: 0x5a10e03e (stale) Mirror 10: 0x5a10e03e (stale) Mirror 11: 0x5a10e03e (stale) Mirror 12: 0x5a10e03e (stale) Mirror 13: 0x5a10e03e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000459 1782481868.319693 write 0x03ba95 thru 0x03ffff (0x456b bytes) 000460 1782481868.331425 mapwrite 0x00b75f thru 0x010e37 (0x56d9 bytes) 000461 1782481868.378008 punch from 0x012d13 to 0x01d0e4 (0xa3d2 bytes) 000462 1782481868.382853 trunc from 0x02b033 to 0x036973 (0xb941 bytes) 000463 1782481868.393172 read 0x027a84 thru 0x027b58 (0xd5 bytes) 000464 1782481868.402743 trunc from 0x02915c to 0x0379ee (0xe893 bytes) 000466 1782481868.414786 trunc from 0x040000 to 0x03f2ed (0xd12 bytes) 000467 1782481868.427855 mapwrite 0x003a26 thru 0x007b3e (0x4119 bytes) 000468 1782481868.448042 punch from 0x015c3f to 0x01a307 (0x46c9 bytes) 000469 1782481868.456104 trunc from 0x016e5f to 0x01eb8f (0x7d31 bytes) 000470 1782481868.462063 trunc from 0x030d17 to 0x03eb7c (0xde66 bytes) 000471 1782481868.466483 read 0x02d65c thru 0x03c23d (0xebe2 bytes) 000472 1782481868.498101 trunc from 0x006df3 to 0x00d656 (0x6864 bytes) 000473 1782481868.509147 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f2ee): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2ee): Mirror 6: 0xe5998e2e (stale) Mirror 8: 0xe5998e2e (stale) Mirror 10: 0xe5998e2e (stale) Mirror 11: 0xe5998e2e (stale) Mirror 12: 0xe5998e2e (stale) Mirror 13: 0xe5998e2e (stale) CRC-32 checksum value for chunk [0, 0x3f2ee): Mirror 1: 0x1055a909 Mirror 14: 0x1055a909 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2ee: skipped 000474 1782481869.362659 read 0x033cc0 thru 0x03589e (0x1bdf bytes) 000476 1782481869.380492 write 0x00e837 thru 0x01e34a (0xfb14 bytes) 000478 1782481869.432860 write 0x004bbc thru 0x00aaa3 (0x5ee8 bytes) 000479 1782481869.442164 write 0x03b867 thru 0x03ffff (0x4799 bytes) 000482 1782481869.450818 read 0x022563 thru 0x02f559 (0xcff7 bytes) 000483 1782481869.470485 read 0x019dd2 thru 0x024c2c (0xae5b bytes) 000484 1782481869.472537 trunc from 0x040000 to 0x01b32e (0x24cd1 bytes) 000485 1782481869.513845 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b32f): Mirror 6: 0x2906a8ee (stale) Mirror 8: 0x2906a8ee (stale) Mirror 10: 0x2906a8ee (stale) Mirror 12: 0x2906a8ee (stale) Mirror 13: 0x2906a8ee (stale) Mirror 14: 0xe1775034 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b32f): Mirror 6: 0x2906a8ee (stale) Mirror 8: 0x2906a8ee (stale) Mirror 10: 0x2906a8ee (stale) Mirror 12: 0x2906a8ee (stale) Mirror 13: 0x2906a8ee (stale) Mirror 14: 0xe1775034 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b32f: skipped 000486 1782481869.876808 mapread 0x00ea86 thru 0x018150 (0x96cb bytes) 000487 1782481869.902806 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b32f): Mirror 8: 0x2906a8ee (stale) Mirror 10: 0x2906a8ee (stale) Mirror 12: 0x2906a8ee (stale) Mirror 13: 0x2906a8ee (stale) Mirror 14: 0xe1775034 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b32f): Mirror 8: 0x2906a8ee (stale) Mirror 10: 0x2906a8ee (stale) Mirror 12: 0x2906a8ee (stale) Mirror 13: 0x2906a8ee (stale) Mirror 14: 0xe1775034 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b32f: skipped 000488 1782481870.218234 trunc from 0x015756 to 0x018490 (0x2d3b bytes) 000489 1782481870.225019 write 0x02dfad thru 0x039c38 (0xbc8c bytes) 000491 1782481870.230823 trunc from 0x039c39 to 0x023cab (0x15f8d bytes) 000492 1782481870.254102 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23cac): Mirror 8: 0xf169db33 (stale) Mirror 10: 0xf169db33 (stale) Mirror 12: 0xf169db33 (stale) Mirror 13: 0xf169db33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23cac): Mirror 8: 0xf169db33 (stale) Mirror 10: 0xf169db33 (stale) Mirror 12: 0xf169db33 (stale) Mirror 13: 0xf169db33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23cac: skipped 000493 1782481870.620384 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23cac): Mirror 8: 0xf169db33 (stale) Mirror 10: 0xf169db33 (stale) Mirror 13: 0xf169db33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23cac): Mirror 8: 0xf169db33 (stale) Mirror 10: 0xf169db33 (stale) Mirror 13: 0xf169db33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23cac: skipped 000494 1782481870.899869 mapwrite 0x00dc14 thru 0x011ccd (0x40ba bytes) 000495 1782481870.942569 fallocate 0x02ad6d thru 0x036c2f (0xbec3 bytes) 000496 1782481870.951648 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36c30): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c30): Mirror 8: 0x1c12be6 (stale) Mirror 10: 0x1c12be6 (stale) Mirror 13: 0x1c12be6 (stale) CRC-32 checksum value for chunk [0, 0x36c30): Mirror 1: 0x24cd2932 Mirror 14: 0x24cd2932 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c30: skipped 000497 1782481871.786661 trunc from 0x036c30 to 0x035f03 (0xd2c bytes) 000498 1782481871.837820 mapread 0x00ad9b thru 0x01aa22 (0xfc88 bytes) 000500 1782481871.867568 mapwrite 0x017bf5 thru 0x025ac1 (0xdecd bytes) 000501 1782481871.894341 mapwrite 0x02a1c4 thru 0x0334da (0x9317 bytes) 000502 1782481871.925485 MIRROR_RESYNC 5 mirrors 000503 1782481873.463437 trunc from 0x035f04 to 0x03eb11 (0x8c0e bytes) 000504 1782481873.501523 trunc from 0x033bd0 to 0x03b01c (0x744d bytes) 000505 1782481873.506409 read 0x007892 thru 0x0141e0 (0xc94f bytes) 000506 1782481873.517503 trunc from 0x008bfc to 0x013160 (0xa565 bytes) 000507 1782481873.523238 mapwrite 0x038e66 thru 0x03ffff (0x719a bytes) 000508 1782481873.580202 mapwrite 0x01e792 thru 0x022508 (0x3d77 bytes) 000509 1782481873.612974 fallocate 0x0007d0 thru 0x007d2a (0x755b bytes) 000510 1782481873.618742 mapread 0x0153fd thru 0x0203e7 (0xafeb bytes) 000511 1782481873.628025 mapwrite 0x03c8ae thru 0x03ffff (0x3752 bytes) 000512 1782481873.641338 read 0x00a924 thru 0x01185e (0x6f3b bytes) 000513 1782481873.651698 write 0x02e496 thru 0x03189b (0x3406 bytes) 000514 1782481873.657400 mapread 0x03032d thru 0x0367df (0x64b3 bytes) 000515 1782481873.659371 mapwrite 0x034597 thru 0x038d29 (0x4793 bytes) 000516 1782481873.678337 write 0x034150 thru 0x03cc49 (0x8afa bytes) 000517 1782481873.679601 trunc from 0x040000 to 0x0189e6 (0x27619 bytes) 000518 1782481873.702690 read 0x00d28a thru 0x00f5e9 (0x2360 bytes) 000519 1782481873.704164 read 0x01293b thru 0x0189e6 (0x60ac bytes) 000520 1782481873.705047 punch from 0x01059f to 0x0189e6 (0x8448 bytes) 000521 1782481873.712533 fallocate 0x01d611 thru 0x024b93 (0x7583 bytes) 000522 1782481873.717205 write 0x00e065 thru 0x01b701 (0xd69d bytes) 000523 1782481873.718974 mapwrite 0x0305a4 thru 0x0326e0 (0x213d bytes) 000525 1782481873.769741 write 0x01a9c1 thru 0x02829f (0xd8df bytes) 000526 1782481873.773672 mapread 0x010a9a thru 0x0181c7 (0x772e bytes) 000528 1782481873.779616 trunc from 0x0326e1 to 0x00a022 (0x286be bytes) 000529 1782481873.806807 trunc from 0x001bc3 to 0x006c2a (0x5068 bytes) 000530 1782481873.817346 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xa023): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa023): Mirror 8: 0x8165cc64 (stale) Mirror 10: 0x8165cc64 (stale) Mirror 13: 0x8165cc64 (stale) Mirror 14: 0x8165cc64 (stale) CRC-32 checksum value for chunk [0, 0xa023): Mirror 1: 0x75da7fa1 Mirror 15: 0x75da7fa1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa023: skipped 000531 1782481874.666276 mapwrite 0x001be4 thru 0x00963b (0x7a58 bytes) 000533 1782481874.815681 fallocate 0x021535 thru 0x02fea2 (0xe96e bytes) 000534 1782481874.823983 trunc from 0x00a023 to 0x03d983 (0x33961 bytes) 000536 1782481874.850896 write 0x008220 thru 0x0164d2 (0xe2b3 bytes) 000537 1782481874.858460 mapread 0x028244 thru 0x03578c (0xd549 bytes) 000538 1782481874.883966 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d984): Mirror 10: 0xd4dd41ea (stale) Mirror 13: 0xd4dd41ea (stale) Mirror 14: 0xd4dd41ea (stale) Mirror 15: 0xa9759147 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d984): Mirror 10: 0xd4dd41ea (stale) Mirror 13: 0xd4dd41ea (stale) Mirror 14: 0xd4dd41ea (stale) Mirror 15: 0xa9759147 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d984: skipped 000539 1782481875.218880 write 0x0197db thru 0x01a06f (0x895 bytes) 000540 1782481875.235450 write 0x0314ea thru 0x0338ac (0x23c3 bytes) 000541 1782481875.243582 trunc from 0x03d984 to 0x023949 (0x1a03a bytes) 000542 1782481875.260434 trunc from 0x002629 to 0x008d7f (0x6757 bytes) 000543 1782481875.268096 fallocate 0x019faa thru 0x023cfd (0x9d54 bytes) 000544 1782481875.275270 mapwrite 0x026b38 thru 0x028657 (0x1b20 bytes) 000545 1782481875.336771 trunc from 0x00342c to 0x00f9e5 (0xc5ba bytes) 000546 1782481875.341792 mapread 0x0085cf thru 0x00f872 (0x72a4 bytes) 000547 1782481875.353456 trunc from 0x028658 to 0x037211 (0xebba bytes) 000548 1782481875.369813 punch from 0x024caa to 0x027b47 (0x2e9e bytes) 000549 1782481875.376658 mapread 0x0189a0 thru 0x021955 (0x8fb6 bytes) 000551 1782481875.392378 mapwrite 0x03f5a5 thru 0x03ffff (0xa5b bytes) 000552 1782481875.442253 mapread 0x02b998 thru 0x034af0 (0x9159 bytes) 000553 1782481875.465410 trunc from 0x040000 to 0x027944 (0x186bb bytes) 000554 1782481875.487543 fallocate 0x00ed14 thru 0x012813 (0x3b00 bytes) 000555 1782481875.494551 fallocate 0x006c43 thru 0x00b41e (0x47dc bytes) 000556 1782481875.501309 write 0x0312d8 thru 0x034980 (0x36a9 bytes) 000558 1782481875.516518 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34981): Mirror 13: 0xf313246e (stale) Mirror 14: 0xf313246e (stale) Mirror 15: 0xbab8b98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34981): Mirror 13: 0xf313246e (stale) Mirror 14: 0xf313246e (stale) Mirror 15: 0xbab8b98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34981: skipped 000560 1782481875.876536 read 0x033088 thru 0x034980 (0x18f9 bytes) 000561 1782481875.883578 mapread 0x020bf4 thru 0x02e613 (0xda20 bytes) 000562 1782481875.938256 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34981): Mirror 13: 0xf313246e (stale) Mirror 14: 0xf313246e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34981): Mirror 13: 0xf313246e (stale) Mirror 14: 0xf313246e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34981: skipped 000563 1782481876.388225 read 0x03156d thru 0x034980 (0x3414 bytes) 000564 1782481876.404326 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34981): Mirror 14: 0xf313246e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34981): Mirror 14: 0xf313246e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34981: skipped 000565 1782481876.761976 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) 000567 1782481876.968926 MIRROR_EXTEND to 2 mirrors 000568 1782481877.717431 fallocate 0x02f90b thru 0x03bfdd (0xc6d3 bytes) 000569 1782481877.757176 trunc from 0x01796f to 0x0199b4 (0x2046 bytes) 000570 1782481877.764086 write 0x00e968 thru 0x018da7 (0xa440 bytes) 000571 1782481877.783587 trunc from 0x03bfde to 0x038a10 (0x35cd bytes) 000572 1782481877.813823 mapread 0x02b037 thru 0x02f5eb (0x45b5 bytes) 000573 1782481877.845271 read 0x02cea5 thru 0x038a10 (0xbb6c bytes) 000574 1782481877.854030 mapwrite 0x02be14 thru 0x031a01 (0x5bee bytes) 000575 1782481877.889433 mapwrite 0x027cf5 thru 0x0290d7 (0x13e3 bytes) 000576 1782481877.912420 mapread 0x00808b thru 0x011c06 (0x9b7c bytes) 000577 1782481877.925521 mapread 0x00e4bc thru 0x01ddf3 (0xf938 bytes) 000578 1782481877.934916 write 0x00d126 thru 0x01846d (0xb348 bytes) 000579 1782481877.940104 MIRROR_RESYNC 2 mirrors 000580 1782481878.525778 mapwrite 0x02dfa0 thru 0x0310b6 (0x3117 bytes) 000581 1782481878.622361 trunc from 0x010465 to 0x01e0e9 (0xdc85 bytes) 000582 1782481878.627354 trunc from 0x038a11 to 0x00f042 (0x299ce bytes) 000583 1782481878.660582 mapwrite 0x03847d thru 0x03ffff (0x7b83 bytes) 000584 1782481878.723821 read 0x01f9c8 thru 0x026564 (0x6b9d bytes) 000585 1782481878.733676 fallocate 0x03fc5a thru 0x03ffff (0x3a6 bytes) 000586 1782481878.740260 trunc from 0x012558 to 0x022030 (0xfad9 bytes) 000587 1782481878.745119 read 0x0264b2 thru 0x02f0ea (0x8c39 bytes) 000588 1782481878.756058 write 0x03cf22 thru 0x03ffff (0x30de bytes) 000589 1782481878.757634 mapwrite 0x0241b4 thru 0x030dc9 (0xcc16 bytes) 000590 1782481878.779029 mapwrite 0x021ed2 thru 0x0304b1 (0xe5e0 bytes) 000591 1782481878.807540 trunc from 0x01f028 to 0x027996 (0x896f bytes) 000592 1782481878.813205 fallocate 0x00156e thru 0x00e967 (0xd3fa bytes) 000593 1782481878.819262 write 0x019e21 thru 0x0232ed (0x94cd bytes) 000594 1782481878.826199 punch from 0x03b256 to 0x03ffff (0x4daa bytes) 000595 1782481878.837404 MIRROR_RESYNC 2 mirrors 000596 1782481879.457535 mapwrite 0x03b3f6 thru 0x03ffff (0x4c0a bytes) 000597 1782481879.527109 trunc from 0x040000 to 0x023bbe (0x1c441 bytes) 000598 1782481879.552143 mapread 0x0044c1 thru 0x00e739 (0xa279 bytes) 000599 1782481879.563603 trunc from 0x023bbf to 0x03547b (0x118bd bytes) 000600 1782481879.579111 fallocate 0x03f89b thru 0x03ffff (0x765 bytes) 000601 1782481879.583694 punch from 0x00e38b to 0x00ee13 (0xa89 bytes) 000602 1782481879.587905 punch from 0x001f30 to 0x00b5ce (0x969f bytes) 000603 1782481879.592683 trunc from 0x03547c to 0x01b15a (0x1a321 bytes) 000605 1782481879.607092 write 0x023b79 thru 0x023e1e (0x2a6 bytes) 000606 1782481879.614866 MIRROR_RESYNC 2 mirrors 000607 1782481880.220604 write 0x017422 thru 0x01e7e8 (0x73c7 bytes) 000608 1782481880.248184 fallocate 0x032bea thru 0x033f5c (0x1373 bytes) 000609 1782481880.252274 trunc from 0x033f5d to 0x00ae7d (0x290df bytes) 000610 1782481880.273232 write 0x03e5b3 thru 0x03ffff (0x1a4d bytes) 000611 1782481880.274352 mapwrite 0x038c71 thru 0x03ffff (0x738f bytes) 000612 1782481880.300979 punch from 0x03b288 to 0x03b437 (0x1b0 bytes) 000613 1782481880.308921 trunc from 0x027eea to 0x036488 (0xe59f bytes) 000614 1782481880.313116 mapread 0x00f231 thru 0x011878 (0x2648 bytes) 000615 1782481880.321538 trunc from 0x040000 to 0x0300e9 (0xff16 bytes) 000616 1782481880.337586 trunc from 0x012da9 to 0x018fe4 (0x623c bytes) 000617 1782481880.344838 MIRROR_RESYNC 2 mirrors 000619 1782481880.772654 mapwrite 0x00c925 thru 0x013e91 (0x756d bytes) 000620 1782481880.869646 trunc from 0x02f4e3 to 0x0300e9 (0xc07 bytes) 000621 1782481880.874765 read 0x00bcb9 thru 0x0133e1 (0x7729 bytes) 000622 1782481880.875902 mapread 0x019e8e thru 0x01bf40 (0x20b3 bytes) 000623 1782481880.894141 MIRROR_RESYNC 2 mirrors 000624 1782481881.472401 write 0x019704 thru 0x01efc1 (0x58be bytes) 000625 1782481881.512401 read 0x02cb8f thru 0x0300e9 (0x355b bytes) 000626 1782481881.520042 trunc from 0x0300ea to 0x00dbe4 (0x22505 bytes) 000627 1782481881.540607 trunc from 0x00d801 to 0x00dbe4 (0x3e4 bytes) 000628 1782481881.544653 read 0x002d9c thru 0x008a59 (0x5cbe bytes) 000629 1782481881.556515 write 0x03b36a thru 0x03ffff (0x4c96 bytes) 000630 1782481881.561250 MIRROR_RESYNC 2 mirrors 000631 1782481882.056161 mapwrite 0x01a88d thru 0x026643 (0xbdb7 bytes) 000632 1782481882.165602 punch from 0x036511 to 0x03bb8a (0x567a bytes) 000633 1782481882.173306 mapread 0x0228af thru 0x02d476 (0xabc8 bytes) 000634 1782481882.190379 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) 000635 1782481882.313375 mapread 0x005faf thru 0x00ed4f (0x8da1 bytes) 000636 1782481882.342047 read 0x02c407 thru 0x031bed (0x57e7 bytes) 000637 1782481882.350263 punch from 0x011592 to 0x013034 (0x1aa3 bytes) 000639 1782481882.356004 punch from 0x0224ce to 0x028103 (0x5c36 bytes) 000640 1782481882.360984 fallocate 0x021cbd thru 0x02c048 (0xa38c bytes) 000642 1782481882.367519 fallocate 0x00efca thru 0x012c0e (0x3c45 bytes) 000645 1782481882.373944 mapwrite 0x00ccb5 thru 0x011f15 (0x5261 bytes) 000646 1782481882.397908 punch from 0x018bd0 to 0x019eec (0x131d bytes) 000647 1782481882.401997 trunc from 0x040000 to 0x018a2b (0x275d4 bytes) 000648 1782481882.447143 mapread 0x0016f1 thru 0x00a26f (0x8b7f bytes) 000649 1782481882.448396 punch from 0x005101 to 0x013955 (0xe855 bytes) 000650 1782481882.452994 punch from 0x004ef6 to 0x0064e3 (0x15ee bytes) 000651 1782481882.457314 trunc from 0x017d82 to 0x018a2b (0xcaa bytes) 000652 1782481882.461394 write 0x027d8e thru 0x02f0a9 (0x731c bytes) 000654 1782481882.464091 read 0x008f86 thru 0x00a166 (0x11e1 bytes) 000655 1782481882.472603 trunc from 0x02f0aa to 0x02e144 (0xf65 bytes) 000656 1782481882.490148 trunc from 0x026ff8 to 0x02e144 (0x714d bytes) 000658 1782481882.504301 read 0x01a0d1 thru 0x01c301 (0x2231 bytes) 000659 1782481882.505076 trunc from 0x02e145 to 0x037591 (0x944d bytes) 000660 1782481882.524742 MIRROR_EXTEND to 2 mirrors 000661 1782481883.064386 mapread 0x001b29 thru 0x0112b8 (0xf790 bytes) 000662 1782481883.094616 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) 000664 1782481883.283085 punch from 0x019f24 to 0x02879f (0xe87c bytes) 000665 1782481883.296352 trunc from 0x037592 to 0x0219f0 (0x15ba1 bytes) 000666 1782481883.336165 trunc from 0x00383f to 0x00f09c (0xb85e bytes) 000667 1782481883.340976 mapread 0x017493 thru 0x019287 (0x1df5 bytes) 000668 1782481883.358778 trunc from 0x0219f1 to 0x00b793 (0x1625d bytes) 000669 1782481883.374204 fallocate 0x00c044 thru 0x014253 (0x8210 bytes) 000670 1782481883.379752 punch from 0x00b9cc to 0x014253 (0x8888 bytes) 000671 1782481883.384332 mapread 0x01226d thru 0x014253 (0x1fe7 bytes) 000672 1782481883.397376 mapwrite 0x01047f thru 0x01451c (0x409e bytes) 000673 1782481883.433079 trunc from 0x005c86 to 0x0136f2 (0xda6d bytes) 000674 1782481883.438158 trunc from 0x01451d to 0x011dbf (0x275d bytes) 000675 1782481883.451597 punch from 0x0023d9 to 0x006938 (0x4560 bytes) 000676 1782481883.457152 mapwrite 0x0067ab thru 0x0148c8 (0xe11e bytes) 000677 1782481883.518173 read 0x00b92d thru 0x0148c8 (0x8f9c bytes) 000678 1782481883.519448 mapread 0x006ade thru 0x00c767 (0x5c8a bytes) 000679 1782481883.521565 punch from 0x00852b to 0x0148c8 (0xc39e bytes) 000680 1782481883.533020 MIRROR_EXTEND to 2 mirrors 000681 1782481884.347959 write 0x023160 thru 0x02a326 (0x71c7 bytes) 000683 1782481884.388512 fallocate 0x01f812 thru 0x02c085 (0xc874 bytes) 000684 1782481884.395497 trunc from 0x02a327 to 0x0397db (0xf4b5 bytes) 000685 1782481884.417403 trunc from 0x038c27 to 0x0397db (0xbb5 bytes) 000686 1782481884.422306 trunc from 0x0397dc to 0x0167b3 (0x23028 bytes) 000687 1782481884.435511 write 0x03c6de thru 0x03fa3b (0x335e bytes) 000688 1782481884.438231 trunc from 0x03fa3c to 0x01b015 (0x24a26 bytes) 000689 1782481884.451835 write 0x03ac5b thru 0x03cd3c (0x20e2 bytes) 000690 1782481884.458603 MIRROR_RESYNC 2 mirrors 000691 1782481885.236423 trunc from 0x03cd3d to 0x03823d (0x4aff bytes) 000692 1782481885.332197 trunc from 0x03823e to 0x011b7e (0x266bf bytes) 000693 1782481885.381150 read 0x0047aa thru 0x009c1f (0x5476 bytes) 000694 1782481885.426592 trunc from 0x010723 to 0x011b7e (0x145c bytes) 000695 1782481885.443860 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x11b7f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11b7f): Mirror 2: 0x2d3acc3e (stale) CRC-32 checksum value for chunk [0, 0x11b7f): Mirror 1: 0x2d3acc3e Mirror 3: 0x2d3acc3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11b7f: skipped 000696 1782481886.036136 fallocate 0x012183 thru 0x019df3 (0x7c71 bytes) 000697 1782481886.064385 punch from 0x00d997 to 0x011b7e (0x41e8 bytes) 000698 1782481886.068623 write 0x024c27 thru 0x026a09 (0x1de3 bytes) 000699 1782481886.074499 punch from 0x01315b to 0x022629 (0xf4cf bytes) 000700 1782481886.078843 mapread 0x021087 thru 0x025de1 (0x4d5b bytes) 000701 1782481886.092048 punch from 0x0158e9 to 0x020ea7 (0xb5bf bytes) 000702 1782481886.096628 read 0x00ac41 thru 0x013191 (0x8551 bytes) 000703 1782481886.108558 trunc from 0x01026e to 0x01b615 (0xb3a8 bytes) 000704 1782481886.113835 punch from 0x023b43 to 0x026a09 (0x2ec7 bytes) 000706 1782481886.128402 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26a0a): Mirror 2: 0x9fa9b360 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a0a): Mirror 2: 0x9fa9b360 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a0a: skipped 000707 1782481886.373379 fallocate 0x0078b0 thru 0x014932 (0xd083 bytes) 000708 1782481886.378359 fallocate 0x02e9c5 thru 0x02f423 (0xa5f bytes) 000709 1782481886.383038 write 0x0098f0 thru 0x017a1a (0xe12b bytes) 000710 1782481886.395478 trunc from 0x02f424 to 0x032e3d (0x3a1a bytes) 000711 1782481886.455223 read 0x031984 thru 0x032e3d (0x14ba bytes) 000712 1782481886.464648 trunc from 0x032e3e to 0x036f5f (0x4122 bytes) 000713 1782481886.482211 fallocate 0x01b54f thru 0x01f4c3 (0x3f75 bytes) 000714 1782481886.488453 read 0x033c05 thru 0x036f5f (0x335b bytes) 000715 1782481886.515486 trunc from 0x036f60 to 0x027b4d (0xf412 bytes) 000716 1782481886.528141 punch from 0x01f063 to 0x022bc2 (0x3b60 bytes) 000717 1782481886.537365 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) 000718 1782481886.666618 punch from 0x01b694 to 0x01fde5 (0x4752 bytes) 000719 1782481886.670096 fallocate 0x0071d1 thru 0x00e0d5 (0x6f05 bytes) 000720 1782481886.673851 trunc from 0x027b4e to 0x01e245 (0x9908 bytes) 000722 1782481886.694020 mapwrite 0x0296ca thru 0x02d971 (0x42a8 bytes) 000723 1782481886.743496 trunc from 0x02b5dc to 0x02d971 (0x2396 bytes) 000724 1782481886.749074 fallocate 0x020885 thru 0x020d09 (0x485 bytes) 000725 1782481886.759188 punch from 0x02547f to 0x02d971 (0x84f3 bytes) 000727 1782481886.766238 write 0x00d33f thru 0x01731a (0x9fdc bytes) 000728 1782481886.782049 mapread 0x0289e1 thru 0x02d971 (0x4f91 bytes) 000729 1782481886.794115 trunc from 0x02d972 to 0x00beda (0x21a97 bytes) 000730 1782481886.820885 write 0x01367b thru 0x02070e (0xd094 bytes) 000731 1782481886.822551 read 0x000c1f thru 0x00de55 (0xd237 bytes) 000732 1782481886.836830 read 0x01d167 thru 0x02070e (0x35a8 bytes) 000733 1782481886.837349 trunc from 0x00a025 to 0x012545 (0x8521 bytes) 000734 1782481886.849116 mapwrite 0x029b9f thru 0x02c1cc (0x262e bytes) 000735 1782481886.910358 mapread 0x01e1f3 thru 0x023cf3 (0x5b01 bytes) 000736 1782481886.911455 mapread 0x029c14 thru 0x02a794 (0xb81 bytes) 000737 1782481886.913275 trunc from 0x0042b3 to 0x013e22 (0xfb70 bytes) 000738 1782481886.918161 trunc from 0x02c1cd to 0x02f590 (0x33c4 bytes) 000739 1782481886.935637 punch from 0x017da0 to 0x01bce8 (0x3f49 bytes) 000740 1782481886.941499 fallocate 0x00218b thru 0x008cf2 (0x6b68 bytes) 000742 1782481886.946567 fallocate 0x030ed9 thru 0x0395e6 (0x870e bytes) 000743 1782481886.951953 punch from 0x01377a to 0x01c883 (0x910a bytes) 000744 1782481886.956983 mapread 0x00b7bc thru 0x010018 (0x485d bytes) 000745 1782481886.986850 trunc from 0x0395e7 to 0x0093f2 (0x301f4 bytes) 000746 1782481887.003430 mapread 0x006b5c thru 0x0093f2 (0x2897 bytes) 000747 1782481887.025617 mapwrite 0x01f792 thru 0x02e1bf (0xea2e bytes) 000748 1782481887.122697 punch from 0x012ad1 to 0x01e769 (0xbc99 bytes) 000749 1782481887.127585 mapread 0x017a63 thru 0x025a98 (0xe036 bytes) 000750 1782481887.166007 write 0x037966 thru 0x03d83e (0x5ed9 bytes) 000751 1782481887.167578 punch from 0x0068ef to 0x0099fb (0x310d bytes) 000754 1782481887.173658 fallocate 0x035895 thru 0x03637b (0xae7 bytes) 000756 1782481887.179245 fallocate 0x00676e thru 0x0164c7 (0xfd5a bytes) 000757 1782481887.184046 mapread 0x0245cf thru 0x03060b (0xc03d bytes) 000758 1782481887.192261 trunc from 0x0317cd to 0x03d83e (0xc072 bytes) 000759 1782481887.206312 read 0x01abd4 thru 0x022ae7 (0x7f14 bytes) 000760 1782481887.207125 mapwrite 0x00fc02 thru 0x014598 (0x4997 bytes) 000762 1782481887.253093 read 0x02e9b2 thru 0x02edf4 (0x443 bytes) 000763 1782481887.253782 trunc from 0x001aca to 0x007893 (0x5dca bytes) 000764 1782481887.259069 fallocate 0x022537 thru 0x02ac66 (0x8730 bytes) 000765 1782481887.263642 trunc from 0x03d83f to 0x00aa37 (0x32e07 bytes) 000766 1782481887.278639 mapwrite 0x00b51b thru 0x012ca0 (0x7786 bytes) 000767 1782481887.319929 write 0x03f681 thru 0x03ffff (0x97f bytes) 000768 1782481887.321125 fallocate 0x02078f thru 0x0230b6 (0x2928 bytes) 000771 1782481887.328293 trunc from 0x028105 to 0x02c393 (0x428f bytes) 000772 1782481887.332580 trunc from 0x0271fe to 0x03486d (0xd670 bytes) 000773 1782481887.336970 write 0x0207aa thru 0x021b23 (0x137a bytes) 000774 1782481887.350318 MIRROR_EXTEND to 2 mirrors 000775 1782481887.922415 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) 000776 1782481888.031564 mapwrite 0x01db12 thru 0x025ca2 (0x8191 bytes) 000777 1782481888.110021 mapread 0x008f70 thru 0x00a5e8 (0x1679 bytes) 000778 1782481888.122352 mapread 0x03519c thru 0x03ffff (0xae64 bytes) 000779 1782481888.149443 punch from 0x007cfe to 0x010fd9 (0x92dc bytes) 000780 1782481888.155765 write 0x01f670 thru 0x026250 (0x6be1 bytes) 000781 1782481888.166976 trunc from 0x00f31b to 0x01c06b (0xcd51 bytes) 000783 1782481888.181094 mapwrite 0x021549 thru 0x02b7a4 (0xa25c bytes) 000784 1782481888.244076 write 0x0041a3 thru 0x0119a7 (0xd805 bytes) 000785 1782481888.249410 fallocate 0x03f6ab thru 0x03ffff (0x955 bytes) 000786 1782481888.255882 mapread 0x025825 thru 0x0266c0 (0xe9c bytes) 000787 1782481888.258427 mapwrite 0x0268e0 thru 0x034019 (0xd73a bytes) 000789 1782481888.299585 punch from 0x00c4d4 to 0x01a962 (0xe48f bytes) 000790 1782481888.316655 read 0x00722c thru 0x01409b (0xce70 bytes) 000791 1782481888.331659 trunc from 0x040000 to 0x0178df (0x28720 bytes) 000792 1782481888.360663 fallocate 0x00bfb5 thru 0x01324a (0x7296 bytes) 000794 1782481888.368488 read 0x005689 thru 0x01499e (0xf316 bytes) 000795 1782481888.369925 trunc from 0x001e29 to 0x00c272 (0xa44a bytes) 000796 1782481888.374837 write 0x038325 thru 0x03f5c2 (0x729e bytes) 000797 1782481888.376266 mapread 0x031c75 thru 0x036165 (0x44f1 bytes) 000798 1782481888.391043 mapwrite 0x01c333 thru 0x01e9b4 (0x2682 bytes) 000799 1782481888.407837 MIRROR_EXTEND to 2 mirrors 000801 1782481888.867355 punch from 0x00ace8 to 0x01715e (0xc477 bytes) 000802 1782481888.905887 punch from 0x036490 to 0x03715c (0xccd bytes) 000803 1782481888.914604 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) 000804 1782481889.010151 trunc from 0x03f5c3 to 0x018c03 (0x269bf bytes) 000805 1782481889.039873 trunc from 0x01163e to 0x018c03 (0x75c6 bytes) 000806 1782481889.042791 fallocate 0x01952b thru 0x01b4e3 (0x1fb9 bytes) 000808 1782481889.046659 write 0x006e15 thru 0x014f8e (0xe17a bytes) 000810 1782481889.057874 trunc from 0x01b4e4 to 0x03ed0c (0x23829 bytes) 000813 1782481889.070137 trunc from 0x000885 to 0x00a3ce (0x9b4a bytes) 000814 1782481889.089221 trunc from 0x027b52 to 0x02bcc8 (0x4177 bytes) 000815 1782481889.094012 mapwrite 0x02a497 thru 0x035809 (0xb373 bytes) 000816 1782481889.130824 punch from 0x03c585 to 0x03ed0c (0x2788 bytes) 000817 1782481889.135185 trunc from 0x03ed0d to 0x033981 (0xb38b bytes) 000818 1782481889.149993 write 0x00b8d0 thru 0x00d718 (0x1e49 bytes) 000819 1782481889.151051 mapread 0x00f32f thru 0x0101b0 (0xe82 bytes) 000820 1782481889.152295 mapread 0x0006e8 thru 0x00f4ce (0xede7 bytes) 000822 1782481889.170290 MIRROR_EXTEND to 2 mirrors 000823 1782481889.659607 trunc from 0x033982 to 0x0274b2 (0xc4cf bytes) 000824 1782481889.725235 MIRROR_RESYNC 2 mirrors 000825 1782481890.142483 trunc from 0x0264c3 to 0x0274b2 (0xff0 bytes) 000826 1782481890.176621 mapwrite 0x003ebb thru 0x00b4a1 (0x75e7 bytes) 000827 1782481890.206510 punch from 0x0228b6 to 0x0274b2 (0x4bfd bytes) 000828 1782481890.210619 write 0x010ea0 thru 0x01e0b0 (0xd211 bytes) 000829 1782481890.224302 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x274b3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x274b3): Mirror 2: 0x83a32800 (stale) CRC-32 checksum value for chunk [0, 0x274b3): Mirror 1: 0xd3f2f1ec Mirror 3: 0xd3f2f1ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x274b3: skipped 000830 1782481890.892147 write 0x01c580 thru 0x02c175 (0xfbf6 bytes) 000831 1782481890.949036 mapwrite 0x01bbf5 thru 0x021e79 (0x6285 bytes) 000832 1782481890.967792 read 0x023cbd thru 0x02c175 (0x84b9 bytes) 000834 1782481890.972157 MIRROR_RESYNC 3 mirrors 000835 1782481891.524964 write 0x00b73d thru 0x01623e (0xab02 bytes) 000836 1782481891.564664 read 0x01ce28 thru 0x021979 (0x4b52 bytes) 000837 1782481891.573529 punch from 0x001a19 to 0x011474 (0xfa5c bytes) 000838 1782481891.588356 trunc from 0x02c176 to 0x01ebf5 (0xd580 bytes) 000839 1782481891.615938 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ebf6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ebf6): Mirror 2: 0x33bc0cea (stale) Mirror 3: 0x33bc0cea (stale) CRC-32 checksum value for chunk [0, 0x1ebf6): Mirror 1: 0x99838aef Mirror 4: 0x99838aef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ebf6: skipped 000840 1782481892.503779 mapread 0x00feb1 thru 0x01c871 (0xc9c1 bytes) 000841 1782481892.532085 punch from 0x0021aa to 0x006b0b (0x4962 bytes) 000842 1782481892.560152 fallocate 0x0357e7 thru 0x03ffff (0xa819 bytes) 000843 1782481892.564149 fallocate 0x02998e thru 0x030956 (0x6fc9 bytes) 000844 1782481892.568758 write 0x011c8d thru 0x015c26 (0x3f9a bytes) 000845 1782481892.579046 mapwrite 0x02c62a thru 0x02f950 (0x3327 bytes) 000846 1782481892.626188 punch from 0x010ab9 to 0x0180f4 (0x763c bytes) 000847 1782481892.631991 mapread 0x0093c9 thru 0x00ea4a (0x5682 bytes) 000848 1782481892.642135 mapread 0x001f4e thru 0x00f2c6 (0xd379 bytes) 000850 1782481892.648990 trunc from 0x02f951 to 0x03f1cc (0xf87c bytes) 000851 1782481892.675871 write 0x03efd1 thru 0x03ffff (0x102f bytes) 000852 1782481892.683678 mapwrite 0x025174 thru 0x02774f (0x25dc bytes) 000853 1782481892.697426 read 0x0007ae thru 0x00d8c1 (0xd114 bytes) 000854 1782481892.707127 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x43df713a (stale) Mirror 3: 0x43df713a (stale) Mirror 4: 0x9e24bad5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x100da580 Mirror 5: 0x100da580 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000855 1782481893.446342 punch from 0x0367b3 to 0x03ffff (0x984d bytes) 000856 1782481893.487066 mapread 0x0027a1 thru 0x003729 (0xf89 bytes) 000857 1782481893.510534 mapwrite 0x00283f thru 0x00a3a1 (0x7b63 bytes) 000858 1782481893.584821 trunc from 0x03d598 to 0x03e0d3 (0xb3c bytes) 000859 1782481893.591862 read 0x025e0f thru 0x035672 (0xf864 bytes) 000860 1782481893.602172 punch from 0x0319d0 to 0x03827b (0x68ac bytes) 000861 1782481893.613630 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x43df713a (stale) Mirror 3: 0x43df713a (stale) Mirror 4: 0x9e24bad5 (stale) Mirror 5: 0x100da580 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcb41e2fe Mirror 6: 0xcb41e2fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000862 1782481894.770765 mapwrite 0x027c11 thru 0x0323bf (0xa7af bytes) 000863 1782481894.865261 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x43df713a (stale) Mirror 3: 0x43df713a (stale) Mirror 4: 0x9e24bad5 (stale) Mirror 5: 0x100da580 (stale) Mirror 6: 0xcb41e2fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4d89f2be Mirror 7: 0x4d89f2be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000864 1782481895.661951 MIRROR_RESYNC 7 mirrors 000865 1782481897.203179 mapread 0x035f65 thru 0x03ffff (0xa09b bytes) 000866 1782481897.231877 mapwrite 0x010fe7 thru 0x0173c7 (0x63e1 bytes) 000867 1782481897.305285 mapwrite 0x005157 thru 0x00a20a (0x50b4 bytes) 000868 1782481897.326306 fallocate 0x01a175 thru 0x02866c (0xe4f8 bytes) 000870 1782481897.331420 fallocate 0x0180c4 thru 0x01ba65 (0x39a2 bytes) 000871 1782481897.338018 write 0x007564 thru 0x0078b8 (0x355 bytes) 000872 1782481897.345297 fallocate 0x00178b thru 0x00887d (0x70f3 bytes) 000873 1782481897.354771 punch from 0x01a13d to 0x01c573 (0x2437 bytes) 000874 1782481897.362581 trunc from 0x040000 to 0x02e486 (0x11b79 bytes) 000875 1782481897.387567 mapread 0x01ea0a thru 0x020323 (0x191a bytes) 000876 1782481897.397356 fallocate 0x01746c thru 0x026b17 (0xf6ac bytes) 000877 1782481897.402244 mapwrite 0x022160 thru 0x022b0a (0x9ab bytes) 000878 1782481897.418413 mapwrite 0x02261f thru 0x02ece2 (0xc6c4 bytes) 000879 1782481897.468171 trunc from 0x02ece3 to 0x02cc1a (0x20c8 bytes) 000880 1782481897.483441 trunc from 0x01cdd6 to 0x0262b6 (0x94e1 bytes) 000881 1782481897.492022 mapread 0x01f0c0 thru 0x023e6e (0x4daf bytes) 000882 1782481897.501667 write 0x033938 thru 0x03cd61 (0x942a bytes) 000883 1782481897.503347 read 0x029935 thru 0x0380d2 (0xe79e bytes) 000885 1782481897.512167 trunc from 0x02e288 to 0x037737 (0x94b0 bytes) 000886 1782481897.526877 punch from 0x02dc6a to 0x03642a (0x87c1 bytes) 000888 1782481897.535532 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cd62): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd62): Mirror 2: 0xef394447 (stale) Mirror 3: 0xef394447 (stale) Mirror 4: 0xef394447 (stale) Mirror 5: 0xef394447 (stale) Mirror 6: 0xef394447 (stale) Mirror 7: 0xef394447 (stale) CRC-32 checksum value for chunk [0, 0x3cd62): Mirror 1: 0x627b207e Mirror 8: 0x627b207e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd62: skipped 000889 1782481898.414136 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3cd62): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cd62): Mirror 2: 0xef394447 (stale) Mirror 3: 0xef394447 (stale) Mirror 4: 0xef394447 (stale) Mirror 5: 0xef394447 (stale) Mirror 6: 0xef394447 (stale) Mirror 7: 0xef394447 (stale) CRC-32 checksum value for chunk [0, 0x3cd62): Mirror 1: 0x627b207e Mirror 8: 0x627b207e Mirror 9: 0x627b207e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cd62: skipped 000890 1782481899.186543 trunc from 0x00e624 to 0x013e07 (0x57e4 bytes) 000891 1782481899.220309 punch from 0x02c6a1 to 0x031c86 (0x55e6 bytes) 000892 1782481899.223869 mapwrite 0x0338de thru 0x0353e0 (0x1b03 bytes) 000893 1782481899.243966 trunc from 0x03cd62 to 0x01527a (0x27ae7 bytes) 000894 1782481899.269175 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1527b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1527b): Mirror 2: 0xf9789d9 (stale) Mirror 3: 0xf9789d9 (stale) Mirror 4: 0xf9789d9 (stale) Mirror 5: 0xf9789d9 (stale) Mirror 6: 0xf9789d9 (stale) Mirror 7: 0xf9789d9 (stale) Mirror 8: 0x6dbe6cba (stale) Mirror 9: 0x6dbe6cba (stale) CRC-32 checksum value for chunk [0, 0x1527b): Mirror 1: 0x7ebbc335 Mirror 10: 0x7ebbc335 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1527b: skipped 000895 1782481900.084451 read 0x00d28e thru 0x01527a (0x7fed bytes) 000896 1782481900.103603 fallocate 0x03d396 thru 0x03ffff (0x2c6a bytes) 000897 1782481900.146171 mapread 0x02ee6c thru 0x036215 (0x73aa bytes) 000898 1782481900.167490 trunc from 0x011f59 to 0x016fc6 (0x506e bytes) 000899 1782481900.171794 trunc from 0x039467 to 0x03c950 (0x34ea bytes) 000900 1782481900.176172 write 0x0317ed thru 0x03ffff (0xe813 bytes) 000901 1782481900.181632 read 0x010aed thru 0x011285 (0x799 bytes) 000902 1782481900.197437 mapwrite 0x01ff9a thru 0x0206ce (0x735 bytes) 000903 1782481900.212395 fallocate 0x02c8fe thru 0x0326f0 (0x5df3 bytes) 000904 1782481900.217495 read 0x03f534 thru 0x03ffff (0xacc bytes) 000905 1782481900.218385 fallocate 0x00c97a thru 0x010e24 (0x44ab bytes) 000906 1782481900.223153 trunc from 0x0080ab to 0x00af36 (0x2e8c bytes) 000907 1782481900.228389 mapread 0x001876 thru 0x010aa4 (0xf22f bytes) 000908 1782481900.234378 trunc from 0x040000 to 0x03059b (0xfa64 bytes) 000909 1782481900.269822 mapread 0x01c5a5 thru 0x01eee2 (0x293e bytes) 000910 1782481900.271856 mapwrite 0x03feb9 thru 0x03ffff (0x147 bytes) 000911 1782481900.308697 punch from 0x03893e to 0x03ffff (0x76c2 bytes) 000912 1782481900.314469 punch from 0x03a8ee to 0x03ffff (0x5712 bytes) 000913 1782481900.320144 trunc from 0x040000 to 0x00dec4 (0x3213b bytes) 000914 1782481900.333523 trunc from 0x0044e6 to 0x005153 (0xc6e bytes) 000916 1782481900.337078 write 0x038f32 thru 0x03ffff (0x70ce bytes) 000917 1782481900.338194 trunc from 0x03d289 to 0x03d675 (0x3ed bytes) 000918 1782481900.349573 mapwrite 0x025341 thru 0x03280a (0xd4ca bytes) 000919 1782481900.439053 read 0x015a44 thru 0x020eb7 (0xb474 bytes) 000920 1782481900.447639 mapwrite 0x015f51 thru 0x018808 (0x28b8 bytes) 000921 1782481900.461539 trunc from 0x040000 to 0x007b83 (0x3847c bytes) 000923 1782481900.483049 write 0x03b60a thru 0x03ffff (0x49f6 bytes) 000924 1782481900.484484 write 0x02bb08 thru 0x038ab5 (0xcfae bytes) 000925 1782481900.494802 punch from 0x015f7a to 0x01e39e (0x8425 bytes) 000927 1782481900.504353 write 0x0326a7 thru 0x03516d (0x2ac7 bytes) 000928 1782481900.511294 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4d89f2be (stale) Mirror 3: 0x4d89f2be (stale) Mirror 4: 0x4d89f2be (stale) Mirror 5: 0x4d89f2be (stale) Mirror 6: 0x4d89f2be (stale) Mirror 8: 0x5786565c (stale) Mirror 9: 0x5786565c (stale) Mirror 10: 0x710bd079 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4d89f2be (stale) Mirror 3: 0x4d89f2be (stale) Mirror 4: 0x4d89f2be (stale) Mirror 5: 0x4d89f2be (stale) Mirror 6: 0x4d89f2be (stale) Mirror 8: 0x5786565c (stale) Mirror 9: 0x5786565c (stale) Mirror 10: 0x710bd079 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000929 1782481901.061396 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4d89f2be (stale) Mirror 3: 0x4d89f2be (stale) Mirror 4: 0x4d89f2be (stale) Mirror 5: 0x4d89f2be (stale) Mirror 8: 0x5786565c (stale) Mirror 9: 0x5786565c (stale) Mirror 10: 0x710bd079 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4d89f2be (stale) Mirror 3: 0x4d89f2be (stale) Mirror 4: 0x4d89f2be (stale) Mirror 5: 0x4d89f2be (stale) Mirror 8: 0x5786565c (stale) Mirror 9: 0x5786565c (stale) Mirror 10: 0x710bd079 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000931 1782481901.519333 read 0x015ffe thru 0x02523d (0xf240 bytes) 000932 1782481901.525583 mapread 0x004857 thru 0x00de0c (0x95b6 bytes) 000933 1782481901.540481 punch from 0x01af38 to 0x02ae69 (0xff32 bytes) 000934 1782481901.546833 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4d89f2be (stale) Mirror 4: 0x4d89f2be (stale) Mirror 5: 0x4d89f2be (stale) Mirror 8: 0x5786565c (stale) Mirror 9: 0x5786565c (stale) Mirror 10: 0x710bd079 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4d89f2be (stale) Mirror 4: 0x4d89f2be (stale) Mirror 5: 0x4d89f2be (stale) Mirror 8: 0x5786565c (stale) Mirror 9: 0x5786565c (stale) Mirror 10: 0x710bd079 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000935 1782481901.838236 read 0x002c9a thru 0x0128dd (0xfc44 bytes) 000937 1782481901.847169 mapread 0x016079 thru 0x01bb34 (0x5abc bytes) 000938 1782481901.861183 mapread 0x02ed1c thru 0x0367f8 (0x7add bytes) 000939 1782481901.870372 trunc from 0x040000 to 0x033ee3 (0xc11c bytes) 000941 1782481901.889327 fallocate 0x027013 thru 0x0346c2 (0xd6b0 bytes) 000943 1782481901.894682 fallocate 0x0073f0 thru 0x00ccb0 (0x58c1 bytes) 000944 1782481901.899215 write 0x03bf0d thru 0x03ffff (0x40f3 bytes) 000945 1782481901.900462 fallocate 0x00cd81 thru 0x016da4 (0xa024 bytes) 000946 1782481901.906201 punch from 0x0258d8 to 0x027f33 (0x265c bytes) 000947 1782481901.909457 write 0x020887 thru 0x02317a (0x28f4 bytes) 000948 1782481901.913900 read 0x032151 thru 0x032350 (0x200 bytes) 000949 1782481901.914400 mapread 0x024829 thru 0x030b5f (0xc337 bytes) 000951 1782481901.925742 read 0x00ee6f thru 0x014cce (0x5e60 bytes) 000952 1782481901.930403 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4d89f2be (stale) Mirror 4: 0x4d89f2be (stale) Mirror 5: 0x4d89f2be (stale) Mirror 9: 0x5786565c (stale) Mirror 10: 0x710bd079 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4d89f2be (stale) Mirror 4: 0x4d89f2be (stale) Mirror 5: 0x4d89f2be (stale) Mirror 9: 0x5786565c (stale) Mirror 10: 0x710bd079 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000953 1782481902.277762 read 0x01e0b6 thru 0x02ddd9 (0xfd24 bytes) 000954 1782481902.283676 read 0x032b5e thru 0x03dc64 (0xb107 bytes) 000955 1782481902.292389 fallocate 0x02f7cb thru 0x034bb3 (0x53e9 bytes) 000956 1782481902.298457 punch from 0x030c66 to 0x03b5d3 (0xa96e bytes) 000957 1782481902.303172 trunc from 0x040000 to 0x03e9ee (0x1611 bytes) 000958 1782481902.319292 mapread 0x018dfd thru 0x01cf29 (0x412d bytes) 000959 1782481902.320544 read 0x0040f1 thru 0x00d667 (0x9577 bytes) 000960 1782481902.325019 MIRROR_RESYNC 6 mirrors 000961 1782481903.602093 write 0x01d895 thru 0x0235ae (0x5d1a bytes) 000962 1782481903.638233 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x200:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9ef): Mirror 3: 0x29c3f18a (stale) Mirror 4: 0x29c3f18a (stale) Mirror 9: 0x29c3f18a (stale) Mirror 10: 0x29c3f18a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9ef): Mirror 3: 0x29c3f18a (stale) Mirror 4: 0x29c3f18a (stale) Mirror 9: 0x29c3f18a (stale) Mirror 10: 0x29c3f18a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9ef: skipped 000963 1782481903.886981 mapwrite 0x036009 thru 0x03ffff (0x9ff7 bytes) 000964 1782481903.927325 trunc from 0x040000 to 0x03dc3b (0x23c4 bytes) 000965 1782481903.942423 fallocate 0x03dfdb thru 0x03ffff (0x2025 bytes) 000966 1782481903.945775 fallocate 0x004d00 thru 0x0052ff (0x600 bytes) 000967 1782481903.949351 read 0x009235 thru 0x0120a1 (0x8e6d bytes) 000968 1782481903.954926 mapread 0x00b682 thru 0x01760c (0xbf8b bytes) 000969 1782481903.961121 trunc from 0x02e22a to 0x0334d1 (0x52a8 bytes) 000970 1782481903.963823 trunc from 0x01ea89 to 0x02b1c7 (0xc73f bytes) 000971 1782481903.969396 trunc from 0x03dc3c to 0x01750b (0x26730 bytes) 000972 1782481903.978454 fallocate 0x013211 thru 0x0219a5 (0xe795 bytes) 000973 1782481903.982348 trunc from 0x01750c to 0x017dee (0x8e3 bytes) 000974 1782481903.993397 trunc from 0x017def to 0x016325 (0x1ac9 bytes) 000975 1782481904.004433 trunc from 0x00a639 to 0x00ed12 (0x46da bytes) 000976 1782481904.007672 write 0x014d17 thru 0x014ea7 (0x191 bytes) 000977 1782481904.008468 fallocate 0x03575e thru 0x035dac (0x64f bytes) 000978 1782481904.011843 trunc from 0x02d134 to 0x035853 (0x8720 bytes) 000979 1782481904.014952 write 0x01b423 thru 0x02a76c (0xf34a bytes) 000980 1782481904.021149 mapread 0x0189b2 thru 0x0218c4 (0x8f13 bytes) 000981 1782481904.026152 mapwrite 0x013c7b thru 0x01bed8 (0x825e bytes) 000983 1782481904.040325 mapwrite 0x0162d6 thru 0x01b795 (0x54c0 bytes) 000985 1782481904.054419 punch from 0x03062d to 0x035dac (0x5780 bytes) 000986 1782481904.057224 trunc from 0x01b713 to 0x026fdb (0xb8c9 bytes) 000987 1782481904.061416 read 0x021c1a thru 0x02ef32 (0xd319 bytes) 000988 1782481904.082069 punch from 0x01a409 to 0x0283f6 (0xdfee bytes) 000989 1782481904.087080 read 0x031016 thru 0x035dac (0x4d97 bytes) 000990 1782481904.087638 punch from 0x01c331 to 0x01c7e1 (0x4b1 bytes) 000991 1782481904.091851 mapwrite 0x02dd4b thru 0x03416c (0x6422 bytes) 000992 1782481904.106201 mapread 0x019dd9 thru 0x022f19 (0x9141 bytes) 000993 1782481904.125984 MIRROR_RESYNC 5 mirrors 000994 1782481905.391236 fallocate 0x007af2 thru 0x010c4f (0x915e bytes) 000995 1782481905.409646 trunc from 0x035dad to 0x03fbaf (0x9e03 bytes) 000996 1782481905.427973 fallocate 0x0030bd thru 0x006ec0 (0x3e04 bytes) 000997 1782481905.431962 punch from 0x037ae4 to 0x038a24 (0xf41 bytes) 000998 1782481905.435101 mapread 0x01fb3f thru 0x02c519 (0xc9db bytes) 000999 1782481905.457186 write 0x0075bb thru 0x00ec4a (0x7690 bytes) 001000 1782481905.465254 trunc from 0x00e9c0 to 0x0173d3 (0x8a14 bytes) All operations completed A-OK! PASS 70a (75s) == sanity-flr test 71: check mirror extend parent fid ==== 09:51:48 (1782481908) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x52:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (7s) == sanity-flr test 72: check append on FLR file ========== 09:51:55 (1782481915) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120109 s, 17.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802405 s, 13.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (6s) == sanity-flr test 200a: stress test ===================== 09:52:02 (1782481922) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre2 Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4994912 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8125792 .. done Extending file size to 5944576 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8387840 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8254560 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8897152 .. done Extending file size to 8563552 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8911264 .. Extending file size to 9008288 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8593280 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9195680 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61481 61482 61483 61487 61489 done 192.168.202.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre2 (opts:) 192.168.202.158@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (76s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:53:17 (1782481997) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre2 Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6086208 .. Extending file size to 5508064 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8612544 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7880576 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7912288 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9274848 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8932960 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9004544 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63120 63121 63123 63126 63128 failed 192.168.202.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre2 (opts:) 192.168.202.158@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (71s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:54:28 (1782482068) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1268996 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2256 1269048 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24860 3582160 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18900 3588120 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43760 7170280 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1268996 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2256 1269048 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24860 3582160 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18900 3588120 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43760 7170280 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0841198 s, 24.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124738 s, 16.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6720d852 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre2 (opts:) PASS 200c (11s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:54:40 (1782482080) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0982811 s, 21.3 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 09:54:46 (1782482086) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x61:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:54:51 (1782482091) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0601329 s, 34.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:54:56 (1782482096) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0874965 s, 24.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2db:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:55:02 (1782482102) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0622225 s, 33.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:55:31 (1782482131) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg258-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561993 s, 18.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2dc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2dc:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2dd:0x0] } PASS 204d (45s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:56:16 (1782482176) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg258-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0520932 s, 40.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:57:00 (1782482220) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg258-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg258-server: Pool lustre.test_204f created Adding targets to pool oleg258-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg258-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0691726 s, 30.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg258-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg258-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg258-server: Pool lustre.test_204f destroyed PASS 204f (62s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:58:02 (1782482282) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:58:09 (1782482289) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:58:14 (1782482294) oleg258-server: Pool lustre.test_206 created Adding targets to pool oleg258-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg258-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg258-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg258-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg258-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:58:36 (1782482316) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844033 s, 12.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391943 s, 26.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x16aa5bbc Mirror 3: 0x16aa5bbc PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:58:45 (1782482325) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:58:48 (1782482328) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:58:51 (1782482331) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0576324 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2470000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1950000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1950000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1950000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1660000 Read this (4 bytes): BBBB 192.168.202.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:59:19 (1782482359) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0333382 s, 246 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:59:24 (1782482364) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347034 s, 30.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11b:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611264 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-26 09:59:25.000000000 -0400 Modify: 2026-06-26 09:59:25.000000000 -0400 Change: 2026-06-26 09:59:26.000000000 -0400 Birth: 2026-06-26 09:59:25.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:59:30 (1782482370) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272028 s, 38.5 MB/s oleg258-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg258-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg258-server: O/280000401/d26/13a: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:59:51 (1782482391) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.205331 s, 199 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0248723 s, 165 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 26 09:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x85:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 26 09:59 /mnt/lustre/f211.sanity-flr PASS 211 (6s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:59:57 (1782482397) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:00:02 (1782482402) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1630 sec =========== 10:00:09 (1782482409) === sanity-flr: start cleanup 10:00:10 (1782482410) === === sanity-flr: finish cleanup 10:00:16 (1782482416) ===