-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 18:13:47 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg424-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 18:14:07 (1787868847) === oleg424-client.virtnet: executing check_config_client /mnt/lustre oleg424-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg424-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff94c998940000.idle_timeout=debug osc.lustre-OST0001-osc-ffff94c998940000.idle_timeout=debug disable quota as required oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 18:14:52 (1787868892) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8218] == sanity-flr test 0a: lfs mirror create with -N option == 18:14:55 (1787868895) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:15:14 (1787868914) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:15:18 (1787868918) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:15:22 (1787868922) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:15:49 (1787868949) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:15:53 (1787868953) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 18:15:57 (1787868957) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff94c998940000.stats=clear osc.lustre-OST0001-osc-ffff94c998940000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:16:07 (1787868967) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:16:19 (1787868979) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:16:28 (1787868988) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:16:32 (1787868992) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:16:41 (1787869001) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 18:16:52 (1787869012) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:16:59 (1787869019) PASS 3 (10s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:17:09 (1787869029) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:17:18 (1787869038) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:17:29 (1787869049) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:17:36 (1787869056) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.422276 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11962 s, 8.8 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:17:45 (1787869065) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.389728 s, 32.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:17:53 (1787869073) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:18:03 (1787869083) Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94c998940000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94c998940000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg424-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94c998940000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94c998940000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg424-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (137s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:20:20 (1787869220) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.287626 s, 18.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94c998940000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94c998940000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:21:08 (1787869268) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94c998940000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94c998940000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94c998940000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94c998940000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg424-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (89s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:22:38 (1787869358) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.26319 s, 16.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 18:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.982s user 0m0.001s sys 0m0.209s Stopping /mnt/lustre-ost1 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94c998940000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94c998940000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.494s user 0m0.002s sys 0m0.207s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg424-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg424-server oleg424-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94c998940000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94c998940000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m2.067s user 0m0.002s sys 0m0.184s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg424-server: oleg424-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg424-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg424-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94c998940000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94c998940000.ost_server_uuid in FULL state after 0 sec PASS 33b (102s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:24:20 (1787869460) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:24:24 (1787869464) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:24:27 (1787869467) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 18:24:31 (1787869471) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 18:24:39 (1787869479) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.526018 s, 23.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:24:52 (1787869492) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389641 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267205 s, 39.2 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:25:01 (1787869501) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:25:11 (1787869511) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000844778 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00192449 s, 532 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000886092 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00120241 s, 852 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 18:25:22 (1787869522) create test files with size 15735972 .. 68958c88371d859aea1b95dc5d820afc - 8c5ecec869d195746dcf58a9dca76c03 - a7fcdaa2c819706eaa17c7fa024c7bb0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 18:25:51 (1787869551) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:26:10 (1787869570) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:26:19 (1787869579) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132702 s, 7.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867727 s, 12.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (13s) == sanity-flr test 41: lfs mirror resync check =========== 18:26:33 (1787869593) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0829349 s, 25.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.343377 s, 12.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142518 s, 14.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 18:26:53 (1787869613) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 18:26:57 (1787869617) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:27:02 (1787869622) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036564 s, 28.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 18:27:10 (1787869630) llite.lustre-ffff94c990d6b000.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (10s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 18:27:20 (1787869640) llite.lustre-ffff94c990d6b000.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328925 s, 31.9 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025441 s, 41.2 MB/s **verify components PASS 43d (10s) == sanity-flr test 44a: lfs mirror split check =========== 18:27:31 (1787869651) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.277945 s, 11.3 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:27:41 (1787869661) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:27:54 (1787869674) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.521673 s, 20.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0xd:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:28:12 (1787869692) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff94c985521800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.482643 s, 21.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg424-client.virtnet /mnt/lustre (opts:) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff94c9b2b32800.xattr_cache=1 PASS 44e (20s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:28:31 (1787869711) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (50s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:29:21 (1787869761) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:30:03 (1787869803) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 18:30:08 (1787869808) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.225697 s, 13.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.204882 s, 15.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x96df203e (nosync,stale) Mirror 2: 0x96df203e (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5fea1b0d157a10ffe91b87ff6a6dc545 - mirror 1 checksum: 9061c31670f99bc112d569b5f6fe357a - mirror 2 checksum: 5fea1b0d157a10ffe91b87ff6a6dc545 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T18:30:12-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T18:30:12-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5fea1b0d157a10ffe91b87ff6a6dc545 - mirror 1 checksum: 9061c31670f99bc112d569b5f6fe357a - mirror 2 checksum: 9061c31670f99bc112d569b5f6fe357a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T18:30:12-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T18:30:12-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 18:30:21 (1787869821) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337929 s, 31.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (10s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:30:31 (1787869831) Failing mds1 on oleg424-server Stopping /mnt/lustre-mds1 (opts:) on oleg424-server 18:30:44 (1787869844) shut down facet: mds1 facet_host: oleg424-server facet_failover_host: oleg424-server Failover mds1 to oleg424-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg424-server: oleg424-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg424-client: oleg424-server: ssh exited with exit code 1 Started lustre-MDT0000 18:31:17 (1787869877) targets are mounted 18:31:17 (1787869877) facet_failover done oleg424-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669716 s, 15.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:31:38 (1787869898) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0757575 s, 270 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0374746 s, 547 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (14s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:31:52 (1787869912) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0992582 s, 206 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00963659 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:32:04 (1787869924) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0146712 s, 1.1 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:32:13 (1787869933) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.121796 s, 168 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:32:22 (1787869942) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0105671 s, 388 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0469051 s, 87.3 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:32:32 (1787869952) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:32:57 (1787869977) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:33:27 (1787870007) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:33:52 (1787870032) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:34:17 (1787870057) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 18:34:27 (1787870067) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 58860 mirror split pid 58861 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58860 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58861 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 18:35:37 (1787870137) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 140 skipping zero length punch hole skipping zero size read skipping zero size read 000004 1787870141.112700 write 0x00d61e thru 0x01a32c (0xcd0f bytes) zero_range to largest ever: 0x13bfd 000006 1787870141.118623 trunc from 0x00f74a to 0x013bfc (0x44b3 bytes) truncating to largest ever: 0x1ba77 000007 1787870141.161452 trunc from 0x01a32d to 0x01ba76 (0x174a bytes) 000008 1787870141.178326 read 0x006f67 thru 0x010260 (0x92fa bytes) 000009 1787870141.191454 mapread 0x009390 thru 0x00a666 (0x12d7 bytes) 000011 1787870141.195681 trunc from 0x00868b to 0x00ae57 (0x27cd bytes) truncating to largest ever: 0x1de66 000012 1787870141.202129 trunc from 0x01ba77 to 0x01de65 (0x23ef bytes) truncating to largest ever: 0x2f62b 000013 1787870141.221080 trunc from 0x01de66 to 0x02f62a (0x117c5 bytes) 000014 1787870141.245315 fallocate 0x010727 thru 0x0166d5 (0x5faf bytes) 000015 1787870141.251825 mapwrite 0x00474a thru 0x007dfd (0x36b4 bytes) 000016 1787870141.290716 trunc from 0x02a8aa to 0x02b8b2 (0x1009 bytes) 000017 1787870141.298429 punch from 0x02e49a to 0x02f62a (0x1191 bytes) 000019 1787870141.311088 mapwrite 0x03febe thru 0x03ffff (0x142 bytes) 000021 1787870141.375354 read 0x000fe2 thru 0x001ec9 (0xee8 bytes) 000022 1787870141.404553 mapread 0x000f27 thru 0x00c048 (0xb122 bytes) 000024 1787870141.407645 punch from 0x015750 to 0x01a8a8 (0x5159 bytes) 000025 1787870141.414772 mapread 0x0040de thru 0x00e395 (0xa2b8 bytes) 000026 1787870141.416689 write 0x01bf10 thru 0x024b4c (0x8c3d bytes) 000027 1787870141.424405 fallocate 0x000cc1 thru 0x0047e0 (0x3b20 bytes) 000028 1787870141.431462 mapread 0x01799c thru 0x01be94 (0x44f9 bytes) 000029 1787870141.467488 mapwrite 0x00bde1 thru 0x01b12a (0xf34a bytes) 000030 1787870141.540247 mapwrite 0x00b065 thru 0x013e6e (0x8e0a bytes) fallocating to largest ever: 0x35742 000031 1787870141.606564 fallocate 0x02ce9f thru 0x035741 (0x88a3 bytes) 000032 1787870141.620691 write 0x02b5e4 thru 0x02f117 (0x3b34 bytes) 000034 1787870141.622454 trunc from 0x040000 to 0x0184ba (0x27b45 bytes) 000035 1787870141.665542 MIRROR_EXTEND to 2 mirrors 000036 1787870142.580664 trunc from 0x013337 to 0x013d5a (0xa24 bytes) 000037 1787870142.629744 trunc from 0x0184bb to 0x002271 (0x16249 bytes) 000038 1787870142.665861 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000039 1787870142.841011 write 0x03af57 thru 0x03ffff (0x50a9 bytes) 000040 1787870142.851032 fallocate 0x01c82b thru 0x027538 (0xad0e bytes) 000041 1787870142.858056 mapwrite 0x0056e7 thru 0x013625 (0xdf3f bytes) 000042 1787870142.916676 mapread 0x002f75 thru 0x0112e6 (0xe372 bytes) zero_range to largest ever: 0x37915 000043 1787870142.919657 trunc from 0x028a1c to 0x037914 (0xeef9 bytes) 000044 1787870142.924874 mapwrite 0x0178b0 thru 0x018dfb (0x154c bytes) 000047 1787870142.946530 fallocate 0x012dde thru 0x020b40 (0xdd63 bytes) 000048 1787870142.953021 read 0x003e2d thru 0x010600 (0xc7d4 bytes) zero_range to largest ever: 0x40000 000049 1787870142.955737 trunc from 0x03e0ec to 0x03ffff (0x1f14 bytes) 000050 1787870142.961422 trunc from 0x040000 to 0x02cd8d (0x13272 bytes) 000051 1787870142.977116 read 0x01afe0 thru 0x01ff53 (0x4f74 bytes) 000052 1787870142.992427 read 0x024e54 thru 0x026916 (0x1ac3 bytes) 000053 1787870142.993204 read 0x00b976 thru 0x0130c3 (0x774e bytes) 000054 1787870142.994360 trunc from 0x02cd8e to 0x01f9d1 (0xd3bc bytes) 000055 1787870143.013840 trunc from 0x0102f9 to 0x012e98 (0x2ba0 bytes) 000056 1787870143.020712 read 0x01f858 thru 0x01f9d1 (0x17a bytes) 000057 1787870143.021730 mapread 0x00ceb4 thru 0x011a68 (0x4bb5 bytes) 000058 1787870143.028501 trunc from 0x0180aa to 0x01f9d1 (0x7928 bytes) 000059 1787870143.036426 mapwrite 0x023352 thru 0x029489 (0x6138 bytes) 000060 1787870143.101087 punch from 0x00b750 to 0x00cef7 (0x17a8 bytes) 000061 1787870143.112157 mapwrite 0x013335 thru 0x01890a (0x55d6 bytes) 000062 1787870143.151850 fallocate 0x002c6d thru 0x00cded (0xa181 bytes) 000063 1787870143.159199 trunc from 0x004b18 to 0x00616f (0x1658 bytes) 000064 1787870143.167537 mapread 0x00f996 thru 0x010bda (0x1245 bytes) 000065 1787870143.170525 write 0x01e4b3 thru 0x02b891 (0xd3df bytes) 000067 1787870143.180053 write 0x0167fe thru 0x025beb (0xf3ee bytes) 000068 1787870143.181872 mapread 0x010f65 thru 0x011dd7 (0xe73 bytes) 000069 1787870143.184462 trunc from 0x0234ef to 0x029858 (0x636a bytes) 000071 1787870143.206531 trunc from 0x02b892 to 0x00d3f4 (0x1e49d bytes) 000072 1787870143.239178 punch from 0x00aa3a to 0x00b7aa (0xd71 bytes) 000073 1787870143.251547 trunc from 0x004bcc to 0x0052eb (0x720 bytes) 000074 1787870143.278800 trunc from 0x00d3f5 to 0x000697 (0xcd5d bytes) 000075 1787870143.308012 trunc from 0x00004b to 0x000697 (0x64d bytes) 000076 1787870143.315479 trunc from 0x0004c4 to 0x000697 (0x1d4 bytes) 000077 1787870143.322409 read 0x00042c thru 0x000697 (0x26c bytes) 000078 1787870143.331620 mapread 0x0000c2 thru 0x000697 (0x5d6 bytes) 000079 1787870143.332620 write 0x015cfe thru 0x018d80 (0x3083 bytes) 000080 1787870143.351045 punch from 0x00f6e9 to 0x010476 (0xd8e bytes) 000081 1787870143.364813 trunc from 0x00605f to 0x009c4c (0x3bee bytes) 000082 1787870143.374151 read 0x01343a thru 0x018d80 (0x5947 bytes) 000083 1787870143.411628 read 0x0122f8 thru 0x018d80 (0x6a89 bytes) 000084 1787870143.420715 read 0x0017ce thru 0x00dbdc (0xc40f bytes) 000085 1787870143.430614 trunc from 0x00f5f9 to 0x018d80 (0x9788 bytes) 000086 1787870143.462955 write 0x03f53e thru 0x03ffff (0xac2 bytes) 000088 1787870143.466077 read 0x017e2b thru 0x0248a7 (0xca7d bytes) 000089 1787870143.481726 punch from 0x0219b7 to 0x0255c2 (0x3c0c bytes) 000090 1787870143.493608 fallocate 0x00eddd thru 0x019f66 (0xb18a bytes) 000091 1787870143.508561 trunc from 0x040000 to 0x0147a7 (0x2b858 bytes) 000093 1787870143.538309 fallocate 0x026146 thru 0x029062 (0x2f1d bytes) 000094 1787870143.547570 fallocate 0x03649f thru 0x03db00 (0x7662 bytes) 000096 1787870143.558354 mapread 0x029454 thru 0x034adf (0xb68c bytes) 000097 1787870143.702753 write 0x025b7f thru 0x032a57 (0xced9 bytes) 000098 1787870143.740872 trunc from 0x0371c0 to 0x03ba5c (0x489d bytes) 000099 1787870143.772801 trunc from 0x03db01 to 0x02c140 (0x119c0 bytes) 000100 1787870143.817548 trunc from 0x009d0d to 0x00f128 (0x541c bytes) 000101 1787870143.826585 write 0x011abc thru 0x01bc53 (0xa198 bytes) 000102 1787870143.844180 trunc from 0x01ccfb to 0x028d66 (0xc06c bytes) 000104 1787870143.887060 mapwrite 0x00c96a thru 0x01aeeb (0xe582 bytes) 000105 1787870143.978076 fallocate 0x023e67 thru 0x02ec46 (0xade0 bytes) 000106 1787870143.997933 mapwrite 0x03e3b6 thru 0x03ffff (0x1c4a bytes) 000107 1787870144.117175 write 0x02ea74 thru 0x0348fe (0x5e8b bytes) 000108 1787870144.155156 mapwrite 0x01d21a thru 0x01f1b5 (0x1f9c bytes) 000109 1787870144.231560 write 0x02d04b thru 0x03b2aa (0xe260 bytes) 000110 1787870144.253436 fallocate 0x0230ec thru 0x032b62 (0xfa77 bytes) 000111 1787870144.260389 trunc from 0x0260e4 to 0x026ffb (0xf18 bytes) 000113 1787870144.283733 write 0x027242 thru 0x034cd5 (0xda94 bytes) 000114 1787870144.296636 mapwrite 0x016503 thru 0x0225c1 (0xc0bf bytes) 000115 1787870144.408029 MIRROR_EXTEND to 2 mirrors 000116 1787870145.524759 fallocate 0x03a9d6 thru 0x03ffff (0x562a bytes) 000118 1787870145.636537 fallocate 0x0143f0 thru 0x020131 (0xbd42 bytes) 000119 1787870145.656818 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000120 1787870145.962924 mapread 0x002d67 thru 0x0036d0 (0x96a bytes) 000121 1787870145.991629 mapread 0x039978 thru 0x03b938 (0x1fc1 bytes) 000123 1787870146.052344 fallocate 0x00c788 thru 0x012d56 (0x65cf bytes) 000124 1787870146.064548 write 0x02d045 thru 0x03810a (0xb0c6 bytes) 000125 1787870146.081119 mapwrite 0x010fbc thru 0x01b529 (0xa56e bytes) 000126 1787870146.124676 fallocate 0x00544d thru 0x00e1e9 (0x8d9d bytes) 000127 1787870146.134160 fallocate 0x013f6c thru 0x020f6e (0xd003 bytes) 000129 1787870146.157176 read 0x002bd5 thru 0x010ae3 (0xdf0f bytes) 000130 1787870146.208049 write 0x008461 thru 0x009168 (0xd08 bytes) 000131 1787870146.218872 mapread 0x03bb17 thru 0x03ffff (0x44e9 bytes) 000132 1787870146.220002 trunc from 0x033499 to 0x03468b (0x11f3 bytes) 000134 1787870146.232367 punch from 0x016e80 to 0x0203ad (0x952e bytes) 000135 1787870146.257317 mapwrite 0x027270 thru 0x02e8ff (0x7690 bytes) 000137 1787870146.324160 write 0x01e647 thru 0x024760 (0x611a bytes) 000138 1787870146.337409 punch from 0x03225f to 0x03411a (0x1ebc bytes) 000139 1787870146.353367 mapread 0x00a4c2 thru 0x017aab (0xd5ea bytes) 000140 1787870146.403303 mapread 0x00630a thru 0x00ef06 (0x8bfd bytes) 000142 1787870146.406858 trunc from 0x00caf3 to 0x00e59b (0x1aa9 bytes) 000144 1787870146.415272 read 0x01ed40 thru 0x0240b2 (0x5373 bytes) 000145 1787870146.415937 read 0x0398c3 thru 0x03ffff (0x673d bytes) 000147 1787870146.419555 trunc from 0x03b0d2 to 0x03ffff (0x4f2e bytes) 000148 1787870146.432110 read 0x03f78f thru 0x03ffff (0x871 bytes) 000151 1787870146.459606 write 0x023faf thru 0x0269c1 (0x2a13 bytes) 000152 1787870146.469482 write 0x0090e8 thru 0x00fe48 (0x6d61 bytes) 000153 1787870146.487036 read 0x021639 thru 0x02ffed (0xe9b5 bytes) 000154 1787870146.487915 read 0x021d99 thru 0x028ecd (0x7135 bytes) 000156 1787870146.491646 mapwrite 0x007f8b thru 0x00ece6 (0x6d5c bytes) 000157 1787870146.552732 trunc from 0x040000 to 0x03bf43 (0x40bc bytes) 000158 1787870146.626344 trunc from 0x03bf44 to 0x003876 (0x386cd bytes) 000159 1787870146.661332 write 0x01738c thru 0x024d17 (0xd98c bytes) 000161 1787870146.668013 mapread 0x004a8f thru 0x005ba9 (0x111b bytes) 000162 1787870146.703124 mapwrite 0x004be4 thru 0x013179 (0xe596 bytes) 000163 1787870146.747674 mapwrite 0x0316bc thru 0x03c46b (0xadb0 bytes) 000165 1787870146.852122 mapwrite 0x02d750 thru 0x02e2fb (0xbac bytes) 000166 1787870146.869538 read 0x001ecd thru 0x008184 (0x62b8 bytes) 000167 1787870146.870390 punch from 0x0221bc to 0x02957b (0x73c0 bytes) 000168 1787870146.878023 read 0x032cc0 thru 0x03c46b (0x97ac bytes) 000169 1787870146.878914 write 0x01fe4a thru 0x02aabd (0xac74 bytes) 000170 1787870146.882407 trunc from 0x011942 to 0x014f56 (0x3615 bytes) 000171 1787870146.888524 fallocate 0x000ec2 thru 0x0079cf (0x6b0e bytes) 000173 1787870146.898357 read 0x03283f thru 0x038728 (0x5eea bytes) 000174 1787870146.900034 read 0x03be82 thru 0x03c46b (0x5ea bytes) 000175 1787870146.901460 mapread 0x0051e5 thru 0x014551 (0xf36d bytes) 000176 1787870146.911251 fallocate 0x0084b1 thru 0x0126ad (0xa1fd bytes) 000177 1787870146.919199 write 0x032179 thru 0x038b4b (0x69d3 bytes) 000178 1787870146.922603 mapread 0x020343 thru 0x0287f6 (0x84b4 bytes) 000179 1787870146.924682 fallocate 0x0151be thru 0x021326 (0xc169 bytes) 000180 1787870146.931473 mapread 0x02acb9 thru 0x030733 (0x5a7b bytes) 000181 1787870146.933670 fallocate 0x019b75 thru 0x0271dc (0xd668 bytes) 000183 1787870146.940637 write 0x0123b6 thru 0x019e4d (0x7a98 bytes) 000184 1787870146.943302 trunc from 0x02ad23 to 0x031590 (0x686e bytes) 000185 1787870146.962964 write 0x01cae0 thru 0x02928f (0xc7b0 bytes) 000186 1787870146.964556 fallocate 0x01561c thru 0x022f01 (0xd8e6 bytes) 000188 1787870146.972022 trunc from 0x03c46c to 0x0262fe (0x1616d bytes) 000189 1787870146.995327 read 0x015901 thru 0x016051 (0x751 bytes) 000190 1787870146.996214 trunc from 0x0162ff to 0x021ad3 (0xb7d5 bytes) 000191 1787870147.020421 mapread 0x0124f2 thru 0x017f4b (0x5a5a bytes) 000192 1787870147.042559 write 0x03ec64 thru 0x03ffff (0x139c bytes) 000193 1787870147.051374 mapwrite 0x03655d thru 0x03ffff (0x9aa3 bytes) 000194 1787870147.098458 mapwrite 0x017e3f thru 0x024c61 (0xce23 bytes) 000195 1787870147.137942 read 0x024375 thru 0x02b009 (0x6c95 bytes) 000196 1787870147.138876 punch from 0x030984 to 0x031921 (0xf9e bytes) 000197 1787870147.146125 trunc from 0x040000 to 0x025b38 (0x1a4c7 bytes) 000199 1787870147.181381 mapwrite 0x0308fb thru 0x03415b (0x3861 bytes) 000200 1787870147.264434 trunc from 0x01cc05 to 0x02a7d4 (0xdbd0 bytes) 000201 1787870147.277727 punch from 0x024233 to 0x02f322 (0xb0f0 bytes) 000202 1787870147.290195 trunc from 0x03415c to 0x038f25 (0x4dca bytes) 000203 1787870147.313800 mapwrite 0x01d30c thru 0x02c034 (0xed29 bytes) 000204 1787870147.515286 punch from 0x01eea2 to 0x02b367 (0xc4c6 bytes) 000205 1787870147.539069 MIRROR_EXTEND to 2 mirrors 000206 1787870148.840000 trunc from 0x035090 to 0x038f25 (0x3e96 bytes) 000207 1787870148.909928 punch from 0x0112b2 to 0x01e1cd (0xcf1c bytes) 000209 1787870148.923745 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000210 1787870149.109073 write 0x027283 thru 0x02f596 (0x8314 bytes) 000212 1787870149.135088 mapread 0x027a85 thru 0x02b502 (0x3a7e bytes) 000213 1787870149.137189 fallocate 0x001806 thru 0x00ce12 (0xb60d bytes) 000214 1787870149.155347 MIRROR_EXTEND to 2 mirrors 000215 1787870149.964149 mapwrite 0x00b78f thru 0x012b66 (0x73d8 bytes) 000216 1787870150.063589 punch from 0x00aecd to 0x015868 (0xa99c bytes) 000217 1787870150.070015 punch from 0x0032b3 to 0x0090f9 (0x5e47 bytes) 000218 1787870150.076540 read 0x00f177 thru 0x011355 (0x21df bytes) 000219 1787870150.082965 mapwrite 0x025786 thru 0x0295c4 (0x3e3f bytes) 000220 1787870150.128471 mapread 0x01dcd1 thru 0x02af00 (0xd230 bytes) 000221 1787870150.149025 mapread 0x037a35 thru 0x038f25 (0x14f1 bytes) 000222 1787870150.186949 read 0x0302f6 thru 0x0306f2 (0x3fd bytes) 000223 1787870150.190321 fallocate 0x00ae4b thru 0x00f7e7 (0x499d bytes) 000224 1787870150.219574 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000225 1787870150.406715 trunc from 0x02aa7c to 0x038650 (0xdbd5 bytes) 000226 1787870150.412046 read 0x034494 thru 0x038f25 (0x4a92 bytes) 000228 1787870150.429982 trunc from 0x01045c to 0x01e1fc (0xdda1 bytes) 000230 1787870150.440910 fallocate 0x03d293 thru 0x03ffff (0x2d6d bytes) 000231 1787870150.445808 mapread 0x0063c6 thru 0x00d511 (0x714c bytes) 000232 1787870150.475430 trunc from 0x02753a to 0x02b66c (0x4133 bytes) 000233 1787870150.481896 punch from 0x0066fe to 0x0083e4 (0x1ce7 bytes) 000234 1787870150.486766 mapread 0x037791 thru 0x038f25 (0x1795 bytes) 000235 1787870150.487631 trunc from 0x0353a5 to 0x038f25 (0x3b81 bytes) 000236 1787870150.493942 trunc from 0x038f26 to 0x0090f6 (0x2fe2f bytes) 000238 1787870150.524452 write 0x017c0d thru 0x01c871 (0x4c65 bytes) 000239 1787870150.525741 trunc from 0x01c872 to 0x035049 (0x187d8 bytes) 000240 1787870150.562071 fallocate 0x028756 thru 0x03673d (0xdfe8 bytes) 000241 1787870150.580944 fallocate 0x005210 thru 0x00a8d9 (0x56ca bytes) 000242 1787870150.592093 punch from 0x02df7b to 0x02f2fd (0x1383 bytes) 000244 1787870150.611772 punch from 0x021666 to 0x02f99a (0xe335 bytes) 000245 1787870150.625667 punch from 0x0229ac to 0x02d525 (0xab7a bytes) 000246 1787870150.652660 fallocate 0x02c893 thru 0x03ab2b (0xe299 bytes) 000247 1787870150.672677 punch from 0x032123 to 0x03331d (0x11fb bytes) 000248 1787870150.716953 read 0x000bc1 thru 0x003604 (0x2a44 bytes) 000249 1787870150.730007 trunc from 0x03a90f to 0x03ab2b (0x21d bytes) 000251 1787870150.735521 write 0x01aa59 thru 0x025d6d (0xb315 bytes) 000252 1787870150.743010 read 0x02a18d thru 0x0341dc (0xa050 bytes) 000253 1787870150.743714 punch from 0x02f327 to 0x0390fe (0x9dd8 bytes) 000254 1787870150.752509 trunc from 0x03ab2c to 0x00d90e (0x2d21d bytes) 000255 1787870150.777101 fallocate 0x022e9d thru 0x02f65b (0xc7bf bytes) 000256 1787870150.784087 mapwrite 0x024a4e thru 0x02c3ad (0x7960 bytes) 000257 1787870150.866978 write 0x010475 thru 0x020297 (0xfe23 bytes) 000259 1787870150.885379 fallocate 0x00b766 thru 0x010254 (0x4aef bytes) 000260 1787870150.911673 punch from 0x0222e4 to 0x02b8b5 (0x95d2 bytes) 000262 1787870150.945451 fallocate 0x019fc6 thru 0x01cdd8 (0x2e13 bytes) 000263 1787870150.966756 punch from 0x024b64 to 0x02ad5d (0x61fa bytes) 000264 1787870150.981930 trunc from 0x02f65c to 0x00115f (0x2e4fc bytes) 000267 1787870151.023132 mapread 0x0008d8 thru 0x00115f (0x888 bytes) 000269 1787870151.024997 trunc from 0x001160 to 0x036e6b (0x35d0c bytes) 000270 1787870151.050829 write 0x01bc0e thru 0x025630 (0x9a23 bytes) 000271 1787870151.089565 trunc from 0x0363cb to 0x036e6b (0xaa1 bytes) 000272 1787870151.101817 fallocate 0x0252db thru 0x031f72 (0xcc98 bytes) 000273 1787870151.114957 trunc from 0x0355fc to 0x036e6b (0x1870 bytes) 000275 1787870151.141216 MIRROR_EXTEND to 2 mirrors 000276 1787870152.285969 read 0x00753d thru 0x00e8fa (0x73be bytes) 000277 1787870152.301841 mapread 0x01b8bd thru 0x024961 (0x90a5 bytes) 000278 1787870152.361306 read 0x01d9c2 thru 0x01f2fe (0x193d bytes) 000279 1787870152.365100 mapwrite 0x0397bd thru 0x03ffff (0x6843 bytes) 000280 1787870152.479542 mapwrite 0x02dd49 thru 0x034103 (0x63bb bytes) 000281 1787870152.499850 fallocate 0x028be7 thru 0x030c49 (0x8063 bytes) 000283 1787870152.506158 fallocate 0x01d8c1 thru 0x0235fc (0x5d3c bytes) 000285 1787870152.512580 mapread 0x0237d6 thru 0x024302 (0xb2d bytes) 000287 1787870152.514897 mapread 0x025808 thru 0x02eafa (0x92f3 bytes) 000288 1787870152.516393 trunc from 0x040000 to 0x021f81 (0x1e07e bytes) 000289 1787870152.548996 punch from 0x009b65 to 0x00e8d8 (0x4d74 bytes) 000290 1787870152.575123 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x21f82): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21f82): Mirror 2: 0x756501a5 (stale) CRC-32 checksum value for chunk [0, 0x21f82): Mirror 1: 0x756501a5 Mirror 3: 0x756501a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21f82: skipped 000291 1787870153.877095 mapwrite 0x01c0ec thru 0x01d2d3 (0x11e8 bytes) 000292 1787870154.030014 mapread 0x012a6c thru 0x016b7b (0x4110 bytes) 000293 1787870154.044221 read 0x00b7e5 thru 0x00c946 (0x1162 bytes) 000294 1787870154.048130 mapread 0x00f4a8 thru 0x0103c7 (0xf20 bytes) 000295 1787870154.050108 mapwrite 0x02cb64 thru 0x034a6a (0x7f07 bytes) 000296 1787870154.172536 mapread 0x00b19c thru 0x00ebc0 (0x3a25 bytes) 000297 1787870154.173525 mapread 0x01f368 thru 0x02b376 (0xc00f bytes) 000298 1787870154.184987 mapread 0x004597 thru 0x00f334 (0xad9e bytes) 000299 1787870154.204979 mapread 0x003506 thru 0x0130df (0xfbda bytes) 000300 1787870154.210657 punch from 0x01ec31 to 0x027e30 (0x9200 bytes) 000301 1787870154.228819 write 0x001c6f thru 0x0112ca (0xf65c bytes) 000302 1787870154.235890 trunc from 0x033988 to 0x034a6a (0x10e3 bytes) 000303 1787870154.252440 trunc from 0x023699 to 0x02d5b0 (0x9f18 bytes) 000304 1787870154.262343 punch from 0x01fe9b to 0x02806d (0x81d3 bytes) 000305 1787870154.271362 trunc from 0x00a2ef to 0x00ee10 (0x4b22 bytes) 000306 1787870154.292487 write 0x016b1d thru 0x01cbdb (0x60bf bytes) 000307 1787870154.294523 mapread 0x021f37 thru 0x0316c8 (0xf792 bytes) 000308 1787870154.326651 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34a6b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a6b): Mirror 2: 0x5bd92807 (stale) Mirror 3: 0x5c8d2bcf (stale) CRC-32 checksum value for chunk [0, 0x34a6b): Mirror 1: 0x4704f064 Mirror 4: 0x4704f064 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a6b: skipped 000309 1787870155.661544 trunc from 0x034a6b to 0x01b35b (0x1970f bytes) 000310 1787870155.804105 trunc from 0x01b35c to 0x0047dc (0x16b7f bytes) 000311 1787870155.838713 mapread 0x002e31 thru 0x0047dc (0x19ac bytes) 000316 1787870155.881195 fallocate 0x019ac0 thru 0x019ddd (0x31e bytes) 000317 1787870155.887936 fallocate 0x0206eb thru 0x02455f (0x3e75 bytes) 000318 1787870155.894817 trunc from 0x0012ed to 0x0070a4 (0x5db8 bytes) 000319 1787870155.915729 punch from 0x00e795 to 0x019ddd (0xb649 bytes) 000320 1787870155.937089 MIRROR_RESYNC 4 mirrors 000321 1787870157.566614 mapwrite 0x01a5ef thru 0x01ebd3 (0x45e5 bytes) 000322 1787870157.777090 mapwrite 0x0320bb thru 0x039f4d (0x7e93 bytes) 000323 1787870157.949529 trunc from 0x010e0f to 0x016c1c (0x5e0e bytes) 000324 1787870157.990376 MIRROR_RESYNC 4 mirrors 000325 1787870159.649584 trunc from 0x039f4e to 0x03eb0c (0x4bbf bytes) 000326 1787870159.775120 trunc from 0x03eb0d to 0x02dbca (0x10f42 bytes) 000327 1787870159.846439 punch from 0x017627 to 0x01a60e (0x2fe8 bytes) 000328 1787870159.856711 punch from 0x02c351 to 0x02dbca (0x187a bytes) 000329 1787870159.876747 write 0x02fe5b thru 0x039ae0 (0x9c86 bytes) 000330 1787870159.879005 punch from 0x034995 to 0x039ae0 (0x514c bytes) 000331 1787870159.924685 write 0x03912b thru 0x03ffff (0x6ed5 bytes) 000333 1787870159.942380 mapwrite 0x00d4b9 thru 0x0114be (0x4006 bytes) 000334 1787870160.025061 punch from 0x004687 to 0x00aee1 (0x685b bytes) 000335 1787870160.031878 mapread 0x01f31a thru 0x02ae02 (0xbae9 bytes) 000336 1787870160.072933 trunc from 0x040000 to 0x00878d (0x37872 bytes) 000338 1787870160.091922 trunc from 0x00878e to 0x033e13 (0x2b686 bytes) 000339 1787870160.119791 fallocate 0x002d1f thru 0x01006a (0xd34c bytes) 000340 1787870160.127903 read 0x01694d thru 0x01fe97 (0x954b bytes) 000341 1787870160.136659 punch from 0x0246cb to 0x033e13 (0xf749 bytes) 000342 1787870160.141674 trunc from 0x02839b to 0x02b36d (0x2fd3 bytes) 000343 1787870160.147037 punch from 0x01967e to 0x0281ce (0xeb51 bytes) 000344 1787870160.154098 trunc from 0x00ea9c to 0x017767 (0x8ccc bytes) 000345 1787870160.159923 trunc from 0x016f35 to 0x01df2d (0x6ff9 bytes) 000346 1787870160.165069 fallocate 0x02ca04 thru 0x02d6dd (0xcda bytes) 000347 1787870160.170823 mapread 0x017baa thru 0x0220d1 (0xa528 bytes) 000348 1787870160.200626 write 0x03e33d thru 0x03ffff (0x1cc3 bytes) 000349 1787870160.201812 fallocate 0x03194b thru 0x03f3c8 (0xda7e bytes) 000350 1787870160.207658 mapread 0x02fc74 thru 0x033099 (0x3426 bytes) 000351 1787870160.219778 mapwrite 0x00b494 thru 0x012a5c (0x75c9 bytes) 000352 1787870160.259021 fallocate 0x01cefb thru 0x029ccf (0xcdd5 bytes) 000353 1787870160.264828 punch from 0x023ff0 to 0x027065 (0x3076 bytes) 000354 1787870160.272978 punch from 0x0208c8 to 0x02e9ed (0xe126 bytes) 000355 1787870160.281288 read 0x02def2 thru 0x039594 (0xb6a3 bytes) 000356 1787870160.299522 mapread 0x02004e thru 0x0271f4 (0x71a7 bytes) 000357 1787870160.300502 trunc from 0x02aa16 to 0x03722c (0xc817 bytes) 000359 1787870160.311951 mapread 0x037ddb thru 0x03bb90 (0x3db6 bytes) 000360 1787870160.317197 mapread 0x0301e5 thru 0x03e5b0 (0xe3cc bytes) 000361 1787870160.370294 trunc from 0x033f4d to 0x03ffff (0xc0b3 bytes) 000362 1787870160.378698 read 0x01331d thru 0x01c87b (0x955f bytes) 000363 1787870160.379925 fallocate 0x004198 thru 0x004af4 (0x95d bytes) 000364 1787870160.385677 mapwrite 0x01509e thru 0x0235b8 (0xe51b bytes) 000365 1787870160.416261 mapread 0x02a4a2 thru 0x038d47 (0xe8a6 bytes) 000366 1787870160.449744 mapread 0x0008b4 thru 0x002fd1 (0x271e bytes) 000368 1787870160.452203 write 0x006492 thru 0x009c60 (0x37cf bytes) 000369 1787870160.453992 trunc from 0x033053 to 0x03ffff (0xcfad bytes) 000370 1787870160.471177 MIRROR_RESYNC 4 mirrors 000371 1787870161.741845 read 0x0201c9 thru 0x02c894 (0xc6cc bytes) 000372 1787870161.755178 punch from 0x005aec to 0x00ef1a (0x942f bytes) 000373 1787870161.832726 punch from 0x005524 to 0x00a7b9 (0x5296 bytes) 000374 1787870161.840859 trunc from 0x040000 to 0x008724 (0x378db bytes) 000375 1787870161.879286 mapwrite 0x0104a0 thru 0x01233d (0x1e9e bytes) 000376 1787870161.975899 fallocate 0x00acc3 thru 0x017377 (0xc6b5 bytes) 000377 1787870161.983729 mapwrite 0x03bc99 thru 0x03c3a2 (0x70a bytes) 000378 1787870162.069019 trunc from 0x03c3a3 to 0x00c070 (0x30332 bytes) 000379 1787870162.143079 read 0x007dfe thru 0x00c070 (0x4273 bytes) 000380 1787870162.171098 trunc from 0x007e2d to 0x00c070 (0x4244 bytes) 000381 1787870162.179253 mapread 0x0078b6 thru 0x00b7cc (0x3f17 bytes) 000382 1787870162.225329 read 0x00a833 thru 0x00c070 (0x183e bytes) 000383 1787870162.245257 read 0x004d20 thru 0x00aa31 (0x5d12 bytes) 000384 1787870162.250069 read 0x00800b thru 0x00c070 (0x4066 bytes) 000385 1787870162.251955 write 0x031d60 thru 0x031e5a (0xfb bytes) 000387 1787870162.258158 trunc from 0x031e5b to 0x034d45 (0x2eeb bytes) 000388 1787870162.285850 write 0x0077a1 thru 0x00c2f4 (0x4b54 bytes) 000390 1787870162.297502 read 0x02a910 thru 0x033d03 (0x93f4 bytes) 000391 1787870162.361573 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34d46): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d46): Mirror 2: 0x4120837 (stale) Mirror 3: 0x4120837 (stale) Mirror 4: 0x4120837 (stale) CRC-32 checksum value for chunk [0, 0x34d46): Mirror 1: 0x923ff7e Mirror 5: 0x923ff7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d46: skipped 000392 1787870164.625950 write 0x0111c1 thru 0x017823 (0x6663 bytes) 000394 1787870164.720190 mapwrite 0x019b41 thru 0x021f63 (0x8423 bytes) 000395 1787870164.815062 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34d46): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d46): Mirror 2: 0x4120837 (stale) Mirror 3: 0x4120837 (stale) Mirror 4: 0x4120837 (stale) Mirror 5: 0x923ff7e (stale) CRC-32 checksum value for chunk [0, 0x34d46): Mirror 1: 0x391f2136 Mirror 6: 0x391f2136 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d46: skipped 000396 1787870166.627011 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34d46): Mirror 3: 0x4120837 (stale) Mirror 4: 0x4120837 (stale) Mirror 5: 0x923ff7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d46): Mirror 3: 0x4120837 (stale) Mirror 4: 0x4120837 (stale) Mirror 5: 0x923ff7e (stale) CRC-32 checksum value for chunk [0, 0x34d46): Mirror 1: 0x391f2136 Mirror 6: 0x391f2136 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d46: skipped 000397 1787870167.690398 fallocate 0x03823f thru 0x03ffff (0x7dc1 bytes) 000398 1787870167.742895 trunc from 0x008bc5 to 0x009cfb (0x1137 bytes) 000399 1787870167.749406 mapwrite 0x02b019 thru 0x02b06a (0x52 bytes) 000400 1787870167.826258 read 0x00594e thru 0x00a3ae (0x4a61 bytes) 000401 1787870167.839348 trunc from 0x040000 to 0x03852a (0x7ad5 bytes) 000402 1787870167.867626 write 0x002b79 thru 0x00e6ce (0xbb56 bytes) 000403 1787870167.881970 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3852b): Mirror 3: 0x38e05753 (stale) Mirror 4: 0x38e05753 (stale) Mirror 5: 0x534b3c5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3852b): Mirror 3: 0x38e05753 (stale) Mirror 4: 0x38e05753 (stale) Mirror 5: 0x534b3c5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3852b: skipped 000404 1787870168.420796 MIRROR_RESYNC 4 mirrors 000405 1787870170.767989 trunc from 0x013950 to 0x020b72 (0xd223 bytes) 000406 1787870170.884135 trunc from 0x03852b to 0x02c7f7 (0xbd33 bytes) 000407 1787870170.938943 mapwrite 0x01fc8f thru 0x029958 (0x9cca bytes) 000408 1787870171.004778 punch from 0x023fa7 to 0x028541 (0x459b bytes) 000409 1787870171.020019 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c7f8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c7f8): Mirror 3: 0x8c499ba9 (stale) Mirror 4: 0x8c499ba9 (stale) Mirror 5: 0x8c499ba9 (stale) CRC-32 checksum value for chunk [0, 0x2c7f8): Mirror 1: 0x10f7dcf Mirror 6: 0x10f7dcf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c7f8: skipped 000410 1787870172.893767 write 0x03cbe1 thru 0x03ffff (0x341f bytes) 000411 1787870173.031464 fallocate 0x020ddc thru 0x024df4 (0x4019 bytes) 000412 1787870173.049801 mapread 0x010382 thru 0x01bfd2 (0xbc51 bytes) 000413 1787870173.161066 trunc from 0x040000 to 0x023910 (0x1c6ef bytes) 000414 1787870173.252748 mapwrite 0x009ce4 thru 0x00b08e (0x13ab bytes) 000416 1787870173.299380 mapread 0x013949 thru 0x0210ab (0xd763 bytes) 000417 1787870173.300864 punch from 0x003abc to 0x011543 (0xda88 bytes) 000418 1787870173.343636 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23911): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23911): Mirror 3: 0xba1fee0a (stale) Mirror 4: 0xba1fee0a (stale) Mirror 5: 0xba1fee0a (stale) Mirror 6: 0xbc50163c (stale) CRC-32 checksum value for chunk [0, 0x23911): Mirror 1: 0x878af249 Mirror 7: 0x878af249 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23911: skipped 000419 1787870174.854388 trunc from 0x0215a0 to 0x023910 (0x2371 bytes) 000420 1787870174.947069 read 0x020c5e thru 0x023910 (0x2cb3 bytes) 000421 1787870174.965869 mapread 0x00c58e thru 0x010797 (0x420a bytes) 000422 1787870175.009687 trunc from 0x01eb2b to 0x023910 (0x4de6 bytes) 000423 1787870175.014170 trunc from 0x004f9f to 0x005ed8 (0xf3a bytes) 000424 1787870175.027111 write 0x024180 thru 0x02e728 (0xa5a9 bytes) 000425 1787870175.043878 trunc from 0x0206ff to 0x023124 (0x2a26 bytes) 000426 1787870175.050808 trunc from 0x027f0b to 0x029598 (0x168e bytes) 000427 1787870175.094979 read 0x00b9d5 thru 0x01b88b (0xfeb7 bytes) 000428 1787870175.118458 fallocate 0x02491e thru 0x030bc8 (0xc2ab bytes) 000430 1787870175.128781 trunc from 0x004e8b to 0x00f918 (0xaa8e bytes) 000431 1787870175.152351 read 0x013d17 thru 0x019385 (0x566f bytes) 000432 1787870175.155154 trunc from 0x00b6b3 to 0x011297 (0x5be5 bytes) 000433 1787870175.163410 trunc from 0x00cd1e to 0x0173bd (0xa6a0 bytes) 000435 1787870175.195302 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e729): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e729): Mirror 3: 0xea0adcc1 (stale) Mirror 4: 0xea0adcc1 (stale) Mirror 5: 0xea0adcc1 (stale) Mirror 6: 0xe6e4f6e7 (stale) Mirror 7: 0x7239f6d (stale) CRC-32 checksum value for chunk [0, 0x2e729): Mirror 1: 0x567a9ed2 Mirror 8: 0x567a9ed2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e729: skipped 000437 1787870176.575459 mapwrite 0x009ec1 thru 0x00a5c2 (0x702 bytes) 000439 1787870176.742901 punch from 0x02d169 to 0x02e728 (0x15c0 bytes) 000440 1787870176.753790 fallocate 0x01ec01 thru 0x01f948 (0xd48 bytes) 000441 1787870176.770193 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2e729): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e729): Mirror 3: 0xea0adcc1 (stale) Mirror 4: 0xea0adcc1 (stale) Mirror 5: 0xea0adcc1 (stale) Mirror 6: 0xe6e4f6e7 (stale) Mirror 7: 0x7239f6d (stale) Mirror 8: 0x567a9ed2 (stale) CRC-32 checksum value for chunk [0, 0x2e729): Mirror 1: 0xf3452268 Mirror 9: 0xf3452268 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e729: skipped 000442 1787870178.603491 punch from 0x021dcf to 0x025474 (0x36a6 bytes) 000443 1787870178.657977 trunc from 0x02e729 to 0x009bbd (0x24b6b bytes) 000444 1787870178.685966 trunc from 0x009707 to 0x009bbd (0x4b7 bytes) 000445 1787870178.691857 trunc from 0x009bbe to 0x03d192 (0x335d5 bytes) 000446 1787870178.711671 fallocate 0x002fb1 thru 0x004ce1 (0x1d31 bytes) 000447 1787870178.717043 mapread 0x023dde thru 0x033942 (0xfb65 bytes) 000448 1787870178.751589 mapwrite 0x005bad thru 0x014de4 (0xf238 bytes) 000449 1787870178.798404 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d193): Mirror 3: 0xcd5dcfa9 (stale) Mirror 4: 0xcd5dcfa9 (stale) Mirror 6: 0xc4f0f6ca (stale) Mirror 7: 0xf79ddcf1 (stale) Mirror 8: 0xa87396c5 (stale) Mirror 9: 0xed801673 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d193): Mirror 3: 0xcd5dcfa9 (stale) Mirror 4: 0xcd5dcfa9 (stale) Mirror 6: 0xc4f0f6ca (stale) Mirror 7: 0xf79ddcf1 (stale) Mirror 8: 0xa87396c5 (stale) Mirror 9: 0xed801673 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d193: skipped 000450 1787870179.531960 trunc from 0x00b254 to 0x017078 (0xbe25 bytes) 000451 1787870179.545720 MIRROR_RESYNC 7 mirrors 000452 1787870182.480445 trunc from 0x039d78 to 0x03a17e (0x407 bytes) 000453 1787870182.524712 trunc from 0x03086d to 0x036846 (0x5fda bytes) 000454 1787870182.532980 punch from 0x03a4cc to 0x03d192 (0x2cc7 bytes) 000455 1787870182.541868 trunc from 0x03d193 to 0x00014f (0x3d043 bytes) 000456 1787870182.570384 MIRROR_RESYNC 7 mirrors 000458 1787870183.631756 punch from 0x00002f to 0x00014f (0x121 bytes) 000459 1787870183.674638 fallocate 0x0054af thru 0x01413b (0xec8d bytes) 000460 1787870183.683714 MIRROR_RESYNC 7 mirrors 000461 1787870184.722315 fallocate 0x0288b9 thru 0x0332a5 (0xa9ed bytes) 000462 1787870184.794820 punch from 0x02fff2 to 0x031ef7 (0x1f06 bytes) 000463 1787870184.814938 mapwrite 0x03d34e thru 0x03ffff (0x2cb2 bytes) 000464 1787870184.900197 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x42581dd3 Mirror 10: 0x42581dd3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000465 1787870186.375696 trunc from 0x040000 to 0x005c85 (0x3a37a bytes) 000466 1787870186.524706 mapread 0x001958 thru 0x005c85 (0x432e bytes) 000467 1787870186.560718 trunc from 0x005c86 to 0x0109fa (0xad75 bytes) 000468 1787870186.616890 mapwrite 0x032ed5 thru 0x0363ee (0x351a bytes) 000469 1787870186.687015 read 0x01326f thru 0x015336 (0x20c8 bytes) 000470 1787870186.702489 fallocate 0x020171 thru 0x021e3b (0x1ccb bytes) 000471 1787870186.719465 trunc from 0x004457 to 0x006578 (0x2122 bytes) 000472 1787870186.729650 mapwrite 0x00a3ad thru 0x016197 (0xbdeb bytes) 000473 1787870186.896025 mapread 0x00a6cc thru 0x011595 (0x6eca bytes) 000474 1787870186.899641 mapread 0x01e9b0 thru 0x02be4c (0xd49d bytes) 000475 1787870186.941606 mapwrite 0x006b0c thru 0x014970 (0xde65 bytes) 000476 1787870186.977637 fallocate 0x0100fb thru 0x010b68 (0xa6e bytes) 000477 1787870186.990376 read 0x007590 thru 0x0098ae (0x231f bytes) 000478 1787870187.005155 MIRROR_RESYNC 8 mirrors 000479 1787870189.510471 mapwrite 0x0280b7 thru 0x02856d (0x4b7 bytes) 000480 1787870189.627036 write 0x033be5 thru 0x03e77b (0xab97 bytes) 000481 1787870189.642732 punch from 0x01e9be to 0x02be36 (0xd479 bytes) 000482 1787870189.649537 write 0x02be9d thru 0x02d285 (0x13e9 bytes) 000483 1787870189.663642 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e77c): Mirror 3: 0x937f98c4 (stale) Mirror 4: 0x937f98c4 (stale) Mirror 6: 0x937f98c4 (stale) Mirror 7: 0x937f98c4 (stale) Mirror 8: 0x937f98c4 (stale) Mirror 9: 0x937f98c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e77c): Mirror 3: 0x937f98c4 (stale) Mirror 4: 0x937f98c4 (stale) Mirror 6: 0x937f98c4 (stale) Mirror 7: 0x937f98c4 (stale) Mirror 8: 0x937f98c4 (stale) Mirror 9: 0x937f98c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e77c: skipped 000484 1787870190.317013 punch from 0x00d552 to 0x013543 (0x5ff2 bytes) 000486 1787870190.329170 punch from 0x03255e to 0x035756 (0x31f9 bytes) 000487 1787870190.335752 trunc from 0x00d12a to 0x01cc5b (0xfb32 bytes) 000488 1787870190.347473 write 0x02cfdc thru 0x0346fd (0x7722 bytes) 000489 1787870190.365858 read 0x02a544 thru 0x031d54 (0x7811 bytes) 000490 1787870190.374125 write 0x01a00f thru 0x02983f (0xf831 bytes) 000491 1787870190.392038 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3e77c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e77c): Mirror 3: 0x937f98c4 (stale) Mirror 4: 0x937f98c4 (stale) Mirror 6: 0x937f98c4 (stale) Mirror 7: 0x937f98c4 (stale) Mirror 8: 0x937f98c4 (stale) Mirror 9: 0x937f98c4 (stale) CRC-32 checksum value for chunk [0, 0x3e77c): Mirror 1: 0x98c207dd Mirror 10: 0x98c207dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e77c: skipped 000492 1787870191.876348 mapread 0x032150 thru 0x03c35e (0xa20f bytes) 000493 1787870191.946773 trunc from 0x03e77c to 0x00ed65 (0x2fa16 bytes) 000494 1787870192.129607 read 0x005ec6 thru 0x00ed65 (0x8ea0 bytes) 000495 1787870192.177418 fallocate 0x023792 thru 0x0276ba (0x3f29 bytes) 000496 1787870192.188637 read 0x021161 thru 0x0276ba (0x655a bytes) 000497 1787870192.231314 MIRROR_RESYNC 8 mirrors 000498 1787870194.344096 punch from 0x00a9cf to 0x00ab55 (0x187 bytes) 000499 1787870194.397675 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x276bb): Mirror 3: 0x4cc7bac5 (stale) Mirror 4: 0x4cc7bac5 (stale) Mirror 7: 0x4cc7bac5 (stale) Mirror 8: 0x4cc7bac5 (stale) Mirror 9: 0x4cc7bac5 (stale) Mirror 10: 0x4cc7bac5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x276bb): Mirror 3: 0x4cc7bac5 (stale) Mirror 4: 0x4cc7bac5 (stale) Mirror 7: 0x4cc7bac5 (stale) Mirror 8: 0x4cc7bac5 (stale) Mirror 9: 0x4cc7bac5 (stale) Mirror 10: 0x4cc7bac5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x276bb: skipped 000501 1787870195.017167 mapread 0x013dcd thru 0x01d143 (0x9377 bytes) 000502 1787870195.045653 trunc from 0x0276bb to 0x02aea0 (0x37e6 bytes) 000503 1787870195.089019 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2aea1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aea1): Mirror 3: 0xd6231ddd (stale) Mirror 4: 0xd6231ddd (stale) Mirror 7: 0xd6231ddd (stale) Mirror 8: 0xd6231ddd (stale) Mirror 9: 0xd6231ddd (stale) Mirror 10: 0xd6231ddd (stale) CRC-32 checksum value for chunk [0, 0x2aea1): Mirror 1: 0xc0c8ce83 Mirror 11: 0xc0c8ce83 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aea1: skipped 000504 1787870196.487591 mapwrite 0x023fd1 thru 0x02acdd (0x6d0d bytes) 000505 1787870196.631638 mapread 0x02849e thru 0x02aea0 (0x2a03 bytes) 000506 1787870196.634689 trunc from 0x0065c1 to 0x00c6cc (0x610c bytes) 000507 1787870196.644177 fallocate 0x03966c thru 0x03d6fc (0x4091 bytes) 000509 1787870196.657749 trunc from 0x003317 to 0x0113fe (0xe0e8 bytes) 000510 1787870196.678181 fallocate 0x03e591 thru 0x03fbe8 (0x1658 bytes) 000511 1787870196.689538 read 0x01f01b thru 0x027959 (0x893f bytes) 000512 1787870196.698476 fallocate 0x0274a1 thru 0x02fecd (0x8a2d bytes) 000513 1787870196.714312 trunc from 0x011aed to 0x01fde5 (0xe2f9 bytes) 000514 1787870196.732952 trunc from 0x02fece to 0x0026f9 (0x2d7d4 bytes) 000515 1787870196.815149 write 0x022c90 thru 0x027d20 (0x5091 bytes) 000517 1787870196.828277 fallocate 0x03bb2d thru 0x03ffff (0x44d3 bytes) 000519 1787870196.841939 trunc from 0x027d21 to 0x03da57 (0x15d37 bytes) 000520 1787870196.878344 mapread 0x0049ef thru 0x006bca (0x21dc bytes) 000521 1787870196.893529 mapwrite 0x00d952 thru 0x017af7 (0xa1a6 bytes) 000522 1787870196.960224 mapwrite 0x02f730 thru 0x0322a5 (0x2b76 bytes) 000523 1787870197.008329 mapwrite 0x008095 thru 0x00d690 (0x55fc bytes) 000524 1787870197.086732 mapwrite 0x0350f2 thru 0x03ffff (0xaf0e bytes) 000525 1787870197.175720 read 0x02ff67 thru 0x03625b (0x62f5 bytes) 000526 1787870197.178920 mapwrite 0x006164 thru 0x014c5d (0xeafa bytes) 000527 1787870197.278208 punch from 0x034871 to 0x03ffff (0xb78f bytes) 000529 1787870197.293671 trunc from 0x040000 to 0x024066 (0x1bf99 bytes) 000530 1787870197.325942 write 0x002686 thru 0x00fe9a (0xd815 bytes) 000531 1787870197.330929 punch from 0x011c68 to 0x012b0d (0xea6 bytes) 000532 1787870197.339943 mapwrite 0x0176d1 thru 0x018c59 (0x1589 bytes) 000534 1787870197.360955 trunc from 0x012b10 to 0x0135dd (0xace bytes) 000535 1787870197.366534 read 0x013512 thru 0x020fb8 (0xdaa7 bytes) 000536 1787870197.376619 write 0x00362f thru 0x012a51 (0xf423 bytes) 000537 1787870197.379130 trunc from 0x024067 to 0x00222d (0x21e39 bytes) 000538 1787870197.399634 mapread 0x001596 thru 0x00222d (0xc98 bytes) 000540 1787870197.403078 trunc from 0x001162 to 0x00222d (0x10cc bytes) 000541 1787870197.408654 write 0x01c866 thru 0x01eafc (0x2297 bytes) 000542 1787870197.410912 read 0x00ced3 thru 0x013d21 (0x6e4f bytes) 000543 1787870197.419397 trunc from 0x01eafd to 0x00894d (0x161af bytes) 000544 1787870197.445027 mapread 0x004652 thru 0x008249 (0x3bf8 bytes) 000545 1787870197.446800 trunc from 0x006287 to 0x0080b0 (0x1e2a bytes) 000546 1787870197.462605 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x894e): Mirror 3: 0x57ddd9a8 (stale) Mirror 7: 0x57ddd9a8 (stale) Mirror 8: 0x57ddd9a8 (stale) Mirror 9: 0x57ddd9a8 (stale) Mirror 10: 0x57ddd9a8 (stale) Mirror 11: 0x57ddd9a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x894e): Mirror 3: 0x57ddd9a8 (stale) Mirror 7: 0x57ddd9a8 (stale) Mirror 8: 0x57ddd9a8 (stale) Mirror 9: 0x57ddd9a8 (stale) Mirror 10: 0x57ddd9a8 (stale) Mirror 11: 0x57ddd9a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x894e: skipped 000547 1787870198.142901 mapwrite 0x0250a9 thru 0x030db4 (0xbd0c bytes) 000548 1787870198.298345 fallocate 0x0369fb thru 0x03ffff (0x9605 bytes) 000549 1787870198.307689 mapwrite 0x027760 thru 0x0360c7 (0xe968 bytes) 000550 1787870198.360990 mapwrite 0x0285be thru 0x0322fd (0x9d40 bytes) 000551 1787870198.396589 read 0x0049ee thru 0x00ea37 (0xa04a bytes) 000553 1787870198.411352 trunc from 0x01db83 to 0x02bf16 (0xe394 bytes) 000554 1787870198.417810 trunc from 0x040000 to 0x01dfac (0x22053 bytes) 000555 1787870198.441768 fallocate 0x02409a thru 0x0243f2 (0x359 bytes) 000556 1787870198.451403 trunc from 0x00a3b5 to 0x00d49a (0x30e6 bytes) 000557 1787870198.459777 read 0x018641 thru 0x019661 (0x1021 bytes) 000558 1787870198.473477 trunc from 0x01dfad to 0x030746 (0x1279a bytes) 000559 1787870198.493851 trunc from 0x02940e to 0x030206 (0x6df9 bytes) 000560 1787870198.505655 MIRROR_RESYNC 7 mirrors 000561 1787870200.750218 mapread 0x011f5f thru 0x01d3e1 (0xb483 bytes) 000562 1787870200.786590 trunc from 0x016cd0 to 0x023ea9 (0xd1da bytes) 000563 1787870200.841643 mapread 0x023935 thru 0x0282ec (0x49b8 bytes) 000564 1787870200.868809 mapwrite 0x037e39 thru 0x03f52f (0x76f7 bytes) 000565 1787870200.976737 punch from 0x00c786 to 0x011397 (0x4c12 bytes) 000566 1787870200.995986 punch from 0x03949f to 0x03c762 (0x32c4 bytes) 000568 1787870201.006848 fallocate 0x03f95e thru 0x03ffff (0x6a2 bytes) 000569 1787870201.016054 read 0x005030 thru 0x012471 (0xd442 bytes) 000570 1787870201.026760 punch from 0x0330b5 to 0x03ffff (0xcf4b bytes) 000571 1787870201.032712 mapread 0x03cf52 thru 0x03ffff (0x30ae bytes) 000572 1787870201.050553 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 12: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000573 1787870202.483499 trunc from 0x040000 to 0x02b1e8 (0x14e17 bytes) 000574 1787870202.578496 trunc from 0x02b1e9 to 0x036ace (0xb8e6 bytes) 000575 1787870202.619620 punch from 0x003451 to 0x00c6fb (0x92ab bytes) 000576 1787870202.630461 punch from 0x00c65a to 0x00d67f (0x1026 bytes) 000578 1787870202.657619 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36acf): Mirror 7: 0xe11aa8a4 (stale) Mirror 8: 0xe11aa8a4 (stale) Mirror 9: 0xe11aa8a4 (stale) Mirror 10: 0xe11aa8a4 (stale) Mirror 11: 0xe11aa8a4 (stale) Mirror 12: 0xe11aa8a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36acf): Mirror 7: 0xe11aa8a4 (stale) Mirror 8: 0xe11aa8a4 (stale) Mirror 9: 0xe11aa8a4 (stale) Mirror 10: 0xe11aa8a4 (stale) Mirror 11: 0xe11aa8a4 (stale) Mirror 12: 0xe11aa8a4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36acf: skipped 000579 1787870203.469102 trunc from 0x00a96e to 0x00f35e (0x49f1 bytes) 000580 1787870203.474713 punch from 0x02e060 to 0x036ace (0x8a6f bytes) 000581 1787870203.481358 fallocate 0x037abe thru 0x03c712 (0x4c55 bytes) 000583 1787870203.490578 mapread 0x011bbc thru 0x01ec1c (0xd061 bytes) 000586 1787870203.551488 punch from 0x02c90b to 0x02f4a5 (0x2b9b bytes) 000587 1787870203.558379 mapwrite 0x0246a2 thru 0x028977 (0x42d6 bytes) 000588 1787870203.630388 write 0x01b081 thru 0x025a73 (0xa9f3 bytes) 000589 1787870203.640410 mapwrite 0x0226b1 thru 0x02cb78 (0xa4c8 bytes) 000590 1787870203.696621 mapwrite 0x02b5a9 thru 0x035452 (0x9eaa bytes) 000591 1787870203.766648 trunc from 0x036acf to 0x037693 (0xbc5 bytes) 000592 1787870203.830415 trunc from 0x0209f1 to 0x0260f6 (0x5706 bytes) 000593 1787870203.844197 trunc from 0x037694 to 0x03fc05 (0x8572 bytes) 000595 1787870203.869923 trunc from 0x03fc06 to 0x025024 (0x1abe1 bytes) 000596 1787870203.913399 read 0x001ea7 thru 0x00cec4 (0xb01e bytes) 000597 1787870203.937592 trunc from 0x01fe23 to 0x025024 (0x5202 bytes) 000598 1787870203.950547 punch from 0x00a4f7 to 0x00daf3 (0x35fd bytes) 000599 1787870203.961441 read 0x015120 thru 0x01abce (0x5aaf bytes) 000602 1787870203.967482 mapwrite 0x034b8a thru 0x03d926 (0x8d9d bytes) 000604 1787870204.079736 punch from 0x012672 to 0x019667 (0x6ff6 bytes) 000605 1787870204.091475 mapwrite 0x02f0de thru 0x031afd (0x2a20 bytes) 000606 1787870204.142382 read 0x01f149 thru 0x02656d (0x7425 bytes) 000607 1787870204.145416 read 0x02f4a9 thru 0x03d926 (0xe47e bytes) 000608 1787870204.150568 mapwrite 0x013e74 thru 0x015839 (0x19c6 bytes) 000609 1787870204.222150 fallocate 0x036976 thru 0x03ffff (0x968a bytes) 000610 1787870204.240022 punch from 0x00a4ec to 0x012f42 (0x8a57 bytes) 000611 1787870204.254474 read 0x02e9e7 thru 0x038232 (0x984c bytes) 000612 1787870204.255541 punch from 0x00d6c5 to 0x0165ea (0x8f26 bytes) 000613 1787870204.261117 punch from 0x018f69 to 0x01fdaa (0x6e42 bytes) 000615 1787870204.269127 punch from 0x0305ec to 0x03e672 (0xe087 bytes) 000616 1787870204.277032 punch from 0x02c60a to 0x033923 (0x731a bytes) 000618 1787870204.284195 punch from 0x01ca1f to 0x027ec6 (0xb4a8 bytes) 000619 1787870204.307024 trunc from 0x040000 to 0x03516b (0xae94 bytes) 000620 1787870204.331022 fallocate 0x01bd9b thru 0x023ba9 (0x7e0f bytes) 000621 1787870204.337197 trunc from 0x017482 to 0x01a9f3 (0x3572 bytes) 000622 1787870204.344387 read 0x01ad25 thru 0x028e57 (0xe133 bytes) 000623 1787870204.359020 trunc from 0x023a85 to 0x02cbb7 (0x9133 bytes) 000624 1787870204.364585 write 0x02db79 thru 0x037d6a (0xa1f2 bytes) 000625 1787870204.367849 fallocate 0x039edf thru 0x03ffff (0x6121 bytes) 000627 1787870204.377125 trunc from 0x0224b5 to 0x02c516 (0xa062 bytes) 000628 1787870204.383456 fallocate 0x011dd5 thru 0x015033 (0x325f bytes) 000629 1787870204.390138 read 0x01d8cc thru 0x028752 (0xae87 bytes) 000630 1787870204.404030 trunc from 0x036a0b to 0x037d6a (0x1360 bytes) 000631 1787870204.430995 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37d6b): Mirror 7: 0xc29d9a57 (stale) Mirror 8: 0xc29d9a57 (stale) Mirror 9: 0xc29d9a57 (stale) Mirror 10: 0xc29d9a57 (stale) Mirror 11: 0xc29d9a57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d6b): Mirror 7: 0xc29d9a57 (stale) Mirror 8: 0xc29d9a57 (stale) Mirror 9: 0xc29d9a57 (stale) Mirror 10: 0xc29d9a57 (stale) Mirror 11: 0xc29d9a57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d6b: skipped 000632 1787870205.061219 mapwrite 0x03000b thru 0x0340ca (0x40c0 bytes) 000633 1787870205.136471 fallocate 0x015d40 thru 0x01e85c (0x8b1d bytes) 000634 1787870205.142796 mapwrite 0x01e845 thru 0x0236a5 (0x4e61 bytes) 000635 1787870205.205551 trunc from 0x019d64 to 0x026d77 (0xd014 bytes) 000636 1787870205.220562 fallocate 0x0087b9 thru 0x0168fb (0xe143 bytes) 000638 1787870205.232324 trunc from 0x0225af to 0x026de0 (0x4832 bytes) 000639 1787870205.244083 mapwrite 0x010459 thru 0x012422 (0x1fca bytes) 000640 1787870205.286814 mapwrite 0x00a0f0 thru 0x00a67d (0x58e bytes) 000641 1787870205.310166 trunc from 0x020379 to 0x021870 (0x14f8 bytes) 000642 1787870205.317151 mapwrite 0x01e92a thru 0x026cfe (0x83d5 bytes) 000643 1787870205.363842 write 0x031add thru 0x03514e (0x3672 bytes) 000644 1787870205.373418 punch from 0x017bc0 to 0x01e4be (0x68ff bytes) 000645 1787870205.383199 trunc from 0x037d6b to 0x00ce10 (0x2af5a bytes) 000646 1787870205.404128 mapread 0x0055f9 thru 0x00ce10 (0x7818 bytes) 000647 1787870205.413928 mapread 0x00223a thru 0x00ce10 (0xabd7 bytes) 000648 1787870205.415187 trunc from 0x007859 to 0x00ce10 (0x55b8 bytes) 000649 1787870205.421360 mapread 0x001d7c thru 0x006b18 (0x4d9d bytes) 000651 1787870205.431348 trunc from 0x00ce11 to 0x02e5ce (0x217be bytes) 000652 1787870205.451384 read 0x001bac thru 0x00cbf6 (0xb04b bytes) 000653 1787870205.463344 write 0x02db45 thru 0x03a09f (0xc55b bytes) 000654 1787870205.469665 trunc from 0x03a0a0 to 0x027054 (0x1304b bytes) 000655 1787870205.493930 mapwrite 0x002c9a thru 0x0070db (0x4442 bytes) 000656 1787870205.515207 trunc from 0x027055 to 0x019c3f (0xd415 bytes) 000657 1787870205.551278 mapwrite 0x026f4f thru 0x02d0e3 (0x6195 bytes) 000658 1787870205.659121 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d0e4): Mirror 7: 0x7a6146c0 (stale) Mirror 8: 0x7a6146c0 (stale) Mirror 9: 0x7a6146c0 (stale) Mirror 10: 0x7a6146c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0e4): Mirror 7: 0x7a6146c0 (stale) Mirror 8: 0x7a6146c0 (stale) Mirror 9: 0x7a6146c0 (stale) Mirror 10: 0x7a6146c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0e4: skipped 000659 1787870206.284529 fallocate 0x022a60 thru 0x02c713 (0x9cb4 bytes) 000660 1787870206.291052 mapread 0x02a21e thru 0x02d0e3 (0x2ec6 bytes) 000661 1787870206.314122 punch from 0x00a512 to 0x0155c0 (0xb0af bytes) 000662 1787870206.319396 trunc from 0x02d0e4 to 0x0295d1 (0x3b12 bytes) 000664 1787870206.348011 read 0x001ec3 thru 0x00b08c (0x91ca bytes) 000665 1787870206.359144 write 0x02aded thru 0x034eea (0xa0fe bytes) 000666 1787870206.362066 mapwrite 0x0157da thru 0x022225 (0xca4c bytes) 000668 1787870206.407181 fallocate 0x03926f thru 0x03bc33 (0x29c5 bytes) 000669 1787870206.414028 mapwrite 0x010303 thru 0x012361 (0x205f bytes) 000670 1787870206.430152 mapread 0x034d5a thru 0x034eea (0x191 bytes) 000671 1787870206.431333 trunc from 0x034eeb to 0x0064a9 (0x2ea41 bytes) 000672 1787870206.459317 read 0x004bcc thru 0x0064a9 (0x18de bytes) 000673 1787870206.461079 fallocate 0x019407 thru 0x01d42c (0x4026 bytes) 000674 1787870206.473004 read 0x00b2ee thru 0x016a1d (0xb730 bytes) 000675 1787870206.497784 mapread 0x019836 thru 0x01d42c (0x3bf7 bytes) 000676 1787870206.502619 trunc from 0x01b2ef to 0x01d42c (0x213e bytes) 000677 1787870206.514076 mapread 0x01198d thru 0x01ceaa (0xb51e bytes) 000678 1787870206.545313 read 0x00373a thru 0x006fef (0x38b6 bytes) 000679 1787870206.552056 mapwrite 0x0135c9 thru 0x020187 (0xcbbf bytes) 000680 1787870206.668210 trunc from 0x020188 to 0x0039ca (0x1c7bd bytes) 000681 1787870206.709185 write 0x01c49a thru 0x02a5b2 (0xe119 bytes) 000682 1787870206.714328 write 0x023684 thru 0x03028f (0xcc0c bytes) 000683 1787870206.719300 write 0x0380ac thru 0x03ffff (0x7f54 bytes) 000685 1787870206.722482 trunc from 0x0087b4 to 0x00ee68 (0x66b5 bytes) 000686 1787870206.728930 mapread 0x038db6 thru 0x039c85 (0xed0 bytes) 000687 1787870206.741316 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe6f2a27b Mirror 11: 0xe6f2a27b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000688 1787870208.329201 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe6f2a27b Mirror 11: 0xe6f2a27b Mirror 12: 0xe6f2a27b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000689 1787870209.842675 trunc from 0x0175e1 to 0x022ae7 (0xb507 bytes) 000690 1787870209.923026 write 0x031a73 thru 0x03ffff (0xe58d bytes) 000691 1787870209.961495 punch from 0x036f61 to 0x03d619 (0x66b9 bytes) 000692 1787870210.011433 trunc from 0x040000 to 0x02d722 (0x128dd bytes) 000693 1787870210.102138 mapread 0x00f0aa thru 0x010188 (0x10df bytes) 000694 1787870210.144648 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d723): Mirror 7: 0xccbc2380 (stale) Mirror 9: 0xccbc2380 (stale) Mirror 10: 0xccbc2380 (stale) Mirror 11: 0x6ddb5036 (stale) Mirror 12: 0x6ddb5036 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d723): Mirror 7: 0xccbc2380 (stale) Mirror 9: 0xccbc2380 (stale) Mirror 10: 0xccbc2380 (stale) Mirror 11: 0x6ddb5036 (stale) Mirror 12: 0x6ddb5036 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d723: skipped 000695 1787870210.787079 punch from 0x004219 to 0x00fc80 (0xba68 bytes) 000696 1787870210.798065 mapread 0x01be80 thru 0x022735 (0x68b6 bytes) 000697 1787870210.849535 trunc from 0x02d723 to 0x028b65 (0x4bbd bytes) 000699 1787870210.902657 punch from 0x022cfe to 0x028b65 (0x5e68 bytes) 000700 1787870210.923792 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28b66): Mirror 9: 0x28993958 (stale) Mirror 10: 0x28993958 (stale) Mirror 11: 0x26299933 (stale) Mirror 12: 0x26299933 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28b66): Mirror 9: 0x28993958 (stale) Mirror 10: 0x28993958 (stale) Mirror 11: 0x26299933 (stale) Mirror 12: 0x26299933 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28b66: skipped 000701 1787870211.496785 write 0x032b04 thru 0x03f1b4 (0xc6b1 bytes) 000702 1787870211.507043 write 0x01eb49 thru 0x020509 (0x19c1 bytes) 000703 1787870211.522438 punch from 0x030534 to 0x0311c8 (0xc95 bytes) 000704 1787870211.544970 MIRROR_RESYNC 5 mirrors 000705 1787870214.090456 mapread 0x00b530 thru 0x016e5f (0xb930 bytes) 000706 1787870214.147840 write 0x017a9e thru 0x02638f (0xe8f2 bytes) 000707 1787870214.295038 mapwrite 0x023e1a thru 0x027c3d (0x3e24 bytes) 000709 1787870214.343743 trunc from 0x03f1b5 to 0x004f10 (0x3a2a4 bytes) 000711 1787870214.400618 fallocate 0x03ff72 thru 0x03ffff (0x8e bytes) 000712 1787870214.412462 read 0x002e1d thru 0x004f10 (0x20f4 bytes) 000713 1787870214.438688 mapread 0x003e6e thru 0x004f10 (0x10a3 bytes) 000714 1787870214.439606 read 0x0048c8 thru 0x004f10 (0x649 bytes) 000715 1787870214.446300 read 0x00045d thru 0x004f10 (0x4ab4 bytes) 000716 1787870214.455958 read 0x001564 thru 0x004f10 (0x39ad bytes) 000717 1787870214.456586 mapwrite 0x02e787 thru 0x0339b1 (0x522b bytes) 000718 1787870214.568604 mapwrite 0x02b19d thru 0x0359bc (0xa820 bytes) 000719 1787870214.645391 mapwrite 0x00be9b thru 0x019b0b (0xdc71 bytes) 000720 1787870214.706872 fallocate 0x006972 thru 0x01557e (0xec0d bytes) 000722 1787870214.719283 trunc from 0x0359bd to 0x03745e (0x1aa2 bytes) 000723 1787870214.762902 punch from 0x01215d to 0x0136ec (0x1590 bytes) 000724 1787870214.769762 punch from 0x00a3c8 to 0x01172a (0x7363 bytes) 000726 1787870214.785164 mapread 0x01721b thru 0x01c64d (0x5433 bytes) 000727 1787870214.786801 mapread 0x02cb36 thru 0x03745e (0xa929 bytes) 000728 1787870214.802166 punch from 0x01da9f to 0x0205d1 (0x2b33 bytes) 000729 1787870214.811081 punch from 0x00ce9c to 0x010045 (0x31aa bytes) 000730 1787870214.822693 mapread 0x0201b8 thru 0x0224c1 (0x230a bytes) 000731 1787870214.836840 trunc from 0x03745f to 0x00b73e (0x2bd20 bytes) 000732 1787870214.871431 write 0x01ad0e thru 0x023467 (0x875a bytes) 000733 1787870214.872896 trunc from 0x01f36d to 0x021992 (0x2626 bytes) 000734 1787870214.931597 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23468): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23468): Mirror 9: 0x2c77b49e (stale) Mirror 10: 0x2c77b49e (stale) Mirror 11: 0x2c77b49e (stale) Mirror 12: 0x2c77b49e (stale) CRC-32 checksum value for chunk [0, 0x23468): Mirror 1: 0x6cf0a8e2 Mirror 13: 0x6cf0a8e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23468: skipped 000736 1787870216.381772 write 0x00fbe6 thru 0x01c288 (0xc6a3 bytes) 000737 1787870216.528789 mapread 0x0187db thru 0x01ba3a (0x3260 bytes) 000738 1787870216.531105 mapread 0x017793 thru 0x023467 (0xbcd5 bytes) 000739 1787870216.558594 read 0x01b427 thru 0x01c782 (0x135c bytes) 000740 1787870216.559559 write 0x008e7c thru 0x017a47 (0xebcc bytes) 000741 1787870216.566371 trunc from 0x01adbe to 0x023467 (0x86aa bytes) 000742 1787870216.595543 read 0x01dadb thru 0x023467 (0x598d bytes) 000743 1787870216.613531 mapread 0x0001d3 thru 0x00ce63 (0xcc91 bytes) 000744 1787870216.614888 fallocate 0x00991d thru 0x012116 (0x87fa bytes) 000745 1787870216.626297 mapwrite 0x03322f thru 0x03d888 (0xa65a bytes) 000746 1787870216.713995 fallocate 0x01cfac thru 0x023e2f (0x6e84 bytes) 000747 1787870216.723380 mapwrite 0x018e38 thru 0x028271 (0xf43a bytes) 000748 1787870216.767960 mapread 0x00516d thru 0x00f55f (0xa3f3 bytes) 000749 1787870216.768955 mapread 0x032b13 thru 0x03d888 (0xad76 bytes) 000750 1787870216.772050 mapwrite 0x037fa7 thru 0x03ffff (0x8059 bytes) 000751 1787870216.830764 read 0x03e0b9 thru 0x03f97f (0x18c7 bytes) 000752 1787870216.845101 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xd46805ec (stale) Mirror 10: 0xd46805ec (stale) Mirror 11: 0xd46805ec (stale) Mirror 12: 0xd46805ec (stale) Mirror 13: 0x4d21dcfb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa4a9922f Mirror 14: 0xa4a9922f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000753 1787870218.174813 punch from 0x0231d7 to 0x0294fe (0x6328 bytes) 000754 1787870218.255367 punch from 0x01f3b6 to 0x02869a (0x92e5 bytes) 000755 1787870218.263067 mapwrite 0x0168f1 thru 0x01e1ba (0x78ca bytes) 000756 1787870218.333627 read 0x017e4a thru 0x01a702 (0x28b9 bytes) 000757 1787870218.344332 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xd46805ec (stale) Mirror 10: 0xd46805ec (stale) Mirror 11: 0xd46805ec (stale) Mirror 12: 0xd46805ec (stale) Mirror 13: 0x4d21dcfb (stale) Mirror 14: 0xa4a9922f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8992e43f Mirror 15: 0x8992e43f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000758 1787870219.895578 mapread 0x015df4 thru 0x016628 (0x835 bytes) 000760 1787870219.925400 read 0x018a19 thru 0x020b0a (0x80f2 bytes) 000762 1787870219.940860 fallocate 0x019360 thru 0x01ecd3 (0x5974 bytes) 000763 1787870220.034679 read 0x010717 thru 0x014215 (0x3aff bytes) 000764 1787870220.056164 trunc from 0x032191 to 0x03a6c3 (0x8533 bytes) 000765 1787870220.063537 write 0x02df3a thru 0x0318d7 (0x399e bytes) 000766 1787870220.070044 trunc from 0x040000 to 0x00f8c3 (0x3073c bytes) 000767 1787870220.122075 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf8c4): Mirror 9: 0xef6b4e70 (stale) Mirror 11: 0xef6b4e70 (stale) Mirror 12: 0xef6b4e70 (stale) Mirror 13: 0xef6b4e70 (stale) Mirror 14: 0xa48887f (stale) Mirror 15: 0xa48887f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf8c4): Mirror 9: 0xef6b4e70 (stale) Mirror 11: 0xef6b4e70 (stale) Mirror 12: 0xef6b4e70 (stale) Mirror 13: 0xef6b4e70 (stale) Mirror 14: 0xa48887f (stale) Mirror 15: 0xa48887f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf8c4: skipped 000768 1787870220.959355 read 0x002d6e thru 0x00e967 (0xbbfa bytes) 000769 1787870220.971297 fallocate 0x02e7dc thru 0x036180 (0x79a5 bytes) 000770 1787870220.977569 punch from 0x024d0f to 0x02bdcc (0x70be bytes) 000771 1787870220.986033 read 0x00b221 thru 0x011f13 (0x6cf3 bytes) 000772 1787870221.003017 trunc from 0x036181 to 0x033b70 (0x2610 bytes) 000773 1787870221.044130 punch from 0x018c8a to 0x027da9 (0xf120 bytes) 000774 1787870221.052994 read 0x017712 thru 0x023a32 (0xc321 bytes) 000775 1787870221.069755 punch from 0x023dfc to 0x0253e2 (0x15e7 bytes) 000776 1787870221.077023 fallocate 0x01af0d thru 0x022b59 (0x7c4d bytes) 000777 1787870221.082443 fallocate 0x00d849 thru 0x014c06 (0x73be bytes) 000779 1787870221.094783 mapwrite 0x030477 thru 0x037acc (0x7656 bytes) 000780 1787870221.176791 read 0x0205c8 thru 0x023c4d (0x3686 bytes) 000781 1787870221.188047 mapwrite 0x015c69 thru 0x0181fe (0x2596 bytes) 000782 1787870221.246928 write 0x034fbc thru 0x03b9c2 (0x6a07 bytes) 000784 1787870221.271714 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b9c3): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9c3): Mirror 9: 0x80165cfb (stale) Mirror 11: 0x80165cfb (stale) Mirror 12: 0x80165cfb (stale) Mirror 13: 0x14dff34a (stale) Mirror 14: 0x8a217d91 (stale) Mirror 15: 0xd84bbd1b (stale) CRC-32 checksum value for chunk [0, 0x3b9c3): Mirror 1: 0xd24fa9bb Mirror 16: 0xd24fa9bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9c3: skipped 000785 1787870223.347433 write 0x00a829 thru 0x0132fa (0x8ad2 bytes) 000787 1787870223.460390 trunc from 0x03b9c3 to 0x03471f (0x72a3 bytes) 000788 1787870223.533953 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x34720): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34720): Mirror 9: 0xd651be9b (stale) Mirror 11: 0xd651be9b (stale) Mirror 12: 0xd651be9b (stale) Mirror 13: 0x8ee58537 (stale) Mirror 14: 0x2a350f6c (stale) Mirror 15: 0x319e1c82 (stale) Mirror 16: 0x6645a1b4 (stale) CRC-32 checksum value for chunk [0, 0x34720): Mirror 1: 0xad14cf69 Mirror 17: 0xad14cf69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34720: skipped 000789 1787870225.136113 mapwrite 0x039a65 thru 0x03dfe7 (0x4583 bytes) 000790 1787870225.271912 mapwrite 0x012859 thru 0x017eff (0x56a7 bytes) 000791 1787870225.364985 mapwrite 0x007ba8 thru 0x013b78 (0xbfd1 bytes) 000792 1787870225.423140 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dfe8): Mirror 11: 0x9af8fdb (stale) Mirror 12: 0x9af8fdb (stale) Mirror 13: 0xa4108732 (stale) Mirror 14: 0xa351c31 (stale) Mirror 15: 0x64be24ae (stale) Mirror 16: 0x46aa1f58 (stale) Mirror 17: 0xdb1272b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dfe8): Mirror 11: 0x9af8fdb (stale) Mirror 12: 0x9af8fdb (stale) Mirror 13: 0xa4108732 (stale) Mirror 14: 0xa351c31 (stale) Mirror 15: 0x64be24ae (stale) Mirror 16: 0x46aa1f58 (stale) Mirror 17: 0xdb1272b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dfe8: skipped 000793 1787870226.119984 fallocate 0x02297a thru 0x024047 (0x16ce bytes) 000794 1787870226.127556 trunc from 0x01b311 to 0x024f0f (0x9bff bytes) 000795 1787870226.135771 mapread 0x01980c thru 0x01fb4d (0x6342 bytes) 000796 1787870226.173072 mapread 0x023e8d thru 0x02c183 (0x82f7 bytes) 000797 1787870226.195035 write 0x0399ca thru 0x03ffff (0x6636 bytes) 000798 1787870226.229981 write 0x03939c thru 0x03ffff (0x6c64 bytes) 000799 1787870226.248668 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xd46805ec (stale) Mirror 12: 0xd46805ec (stale) Mirror 13: 0x4d21dcfb (stale) Mirror 14: 0xa4a9922f (stale) Mirror 15: 0x8992e43f (stale) Mirror 16: 0x94bbdf7f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xd46805ec (stale) Mirror 12: 0xd46805ec (stale) Mirror 13: 0x4d21dcfb (stale) Mirror 14: 0xa4a9922f (stale) Mirror 15: 0x8992e43f (stale) Mirror 16: 0x94bbdf7f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000800 1787870226.948141 trunc from 0x040000 to 0x02f129 (0x10ed6 bytes) 000801 1787870226.990719 read 0x021035 thru 0x023d00 (0x2ccc bytes) 000802 1787870227.011875 read 0x011e28 thru 0x01ebae (0xcd87 bytes) 000803 1787870227.031119 mapread 0x0041e4 thru 0x00ecaa (0xaac7 bytes) 000804 1787870227.033133 mapread 0x02b7ee thru 0x02f129 (0x393c bytes) 000805 1787870227.035727 read 0x005f89 thru 0x0069ee (0xa66 bytes) 000806 1787870227.047476 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f12a): Mirror 12: 0x21d96d2b (stale) Mirror 13: 0xa791d092 (stale) Mirror 14: 0xa8e07ff3 (stale) Mirror 15: 0x81b0e77f (stale) Mirror 16: 0x3d4160b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f12a): Mirror 12: 0x21d96d2b (stale) Mirror 13: 0xa791d092 (stale) Mirror 14: 0xa8e07ff3 (stale) Mirror 15: 0x81b0e77f (stale) Mirror 16: 0x3d4160b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f12a: skipped 000807 1787870227.661102 write 0x00e6de thru 0x01992a (0xb24d bytes) 000808 1787870227.692100 mapread 0x0172d5 thru 0x01a540 (0x326c bytes) 000809 1787870227.701506 punch from 0x029d58 to 0x02f129 (0x53d2 bytes) 000811 1787870227.717644 read 0x00a108 thru 0x0194aa (0xf3a3 bytes) 000812 1787870227.734145 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f12a): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f12a): Mirror 12: 0x21d96d2b (stale) Mirror 13: 0xa791d092 (stale) Mirror 14: 0xa8e07ff3 (stale) Mirror 15: 0x81b0e77f (stale) Mirror 16: 0x3d4160b6 (stale) CRC-32 checksum value for chunk [0, 0x2f12a): Mirror 1: 0x7fd18145 Mirror 17: 0x7fd18145 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f12a: skipped 000813 1787870229.511487 trunc from 0x02f12a to 0x011a59 (0x1d6d0 bytes) 000814 1787870229.588501 read 0x00b209 thru 0x011a59 (0x6851 bytes) 000815 1787870229.603541 write 0x00b590 thru 0x01a51b (0xef8c bytes) 000816 1787870229.605558 write 0x0051a9 thru 0x0112b1 (0xc109 bytes) 000817 1787870229.619757 read 0x00c847 thru 0x01197e (0x5138 bytes) 000818 1787870229.638536 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a51c): Mirror 13: 0xab2264d5 (stale) Mirror 14: 0x77098b9d (stale) Mirror 15: 0x7f82e48b (stale) Mirror 16: 0x1e41a3d9 (stale) Mirror 17: 0x1772f1be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a51c): Mirror 13: 0xab2264d5 (stale) Mirror 14: 0x77098b9d (stale) Mirror 15: 0x7f82e48b (stale) Mirror 16: 0x1e41a3d9 (stale) Mirror 17: 0x1772f1be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a51c: skipped 000819 1787870230.449628 write 0x001eb1 thru 0x00c685 (0xa7d5 bytes) 000820 1787870230.472396 fallocate 0x006518 thru 0x00f6b7 (0x91a0 bytes) 000821 1787870230.479907 fallocate 0x030b75 thru 0x03ac36 (0xa0c2 bytes) 000822 1787870230.490080 punch from 0x0075c6 to 0x00ab8d (0x35c8 bytes) 000823 1787870230.522635 mapwrite 0x01685e thru 0x01dc29 (0x73cc bytes) 000825 1787870230.634868 trunc from 0x01dc2a to 0x035a2a (0x17e01 bytes) 000826 1787870230.666320 MIRROR_RESYNC 6 mirrors 000828 1787870233.410066 trunc from 0x035a2b to 0x02d1cf (0x885b bytes) 000829 1787870233.466915 read 0x02bf46 thru 0x02d1cf (0x128a bytes) 000830 1787870233.474889 trunc from 0x01412f to 0x01f6df (0xb5b1 bytes) 000831 1787870233.480075 mapwrite 0x002731 thru 0x00e752 (0xc022 bytes) 000832 1787870233.549577 fallocate 0x02eb83 thru 0x039a77 (0xaef5 bytes) 000833 1787870233.562930 fallocate 0x02ef1c thru 0x03a600 (0xb6e5 bytes) 000834 1787870233.571718 mapread 0x029132 thru 0x02f461 (0x6330 bytes) 000836 1787870233.586662 mapwrite 0x018b87 thru 0x019c7b (0x10f5 bytes) 000838 1787870233.624960 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39a78): Mirror 13: 0x5b8081b5 (stale) Mirror 14: 0x5b8081b5 (stale) Mirror 16: 0x5b8081b5 (stale) Mirror 17: 0x5b8081b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a78): Mirror 13: 0x5b8081b5 (stale) Mirror 14: 0x5b8081b5 (stale) Mirror 16: 0x5b8081b5 (stale) Mirror 17: 0x5b8081b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a78: skipped 000839 1787870234.378200 mapwrite 0x030330 thru 0x03ce92 (0xcb63 bytes) 000840 1787870234.519979 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ce93): Mirror 13: 0xbe92ecca (stale) Mirror 16: 0xbe92ecca (stale) Mirror 17: 0xbe92ecca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce93): Mirror 13: 0xbe92ecca (stale) Mirror 16: 0xbe92ecca (stale) Mirror 17: 0xbe92ecca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce93: skipped 000842 1787870235.339664 write 0x03f1b8 thru 0x03ffff (0xe48 bytes) 000843 1787870235.347585 mapwrite 0x00ef76 thru 0x01339e (0x4429 bytes) 000844 1787870235.431482 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xbe2711d3 (stale) Mirror 17: 0xbe2711d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xbe2711d3 (stale) Mirror 17: 0xbe2711d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000845 1787870235.845216 fallocate 0x033ddf thru 0x03651d (0x273f bytes) 000846 1787870235.857917 trunc from 0x040000 to 0x02dffb (0x12004 bytes) 000847 1787870235.890505 write 0x01c0d2 thru 0x028683 (0xc5b2 bytes) 000848 1787870235.924915 trunc from 0x02dffc to 0x00211e (0x2bedd bytes) 000849 1787870235.948186 mapread 0x001d8b thru 0x00211e (0x394 bytes) 000850 1787870235.974300 write 0x00ba97 thru 0x013937 (0x7ea1 bytes) 000851 1787870235.976020 mapwrite 0x00d55e thru 0x00fbfd (0x26a0 bytes) 000852 1787870236.031785 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13938): Mirror 16: 0xaae40747 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13938): Mirror 16: 0xaae40747 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13938: skipped 000853 1787870236.590834 write 0x03432d thru 0x03ce38 (0x8b0c bytes) 000854 1787870236.603761 trunc from 0x01d03e to 0x02161a (0x45dd bytes) 000855 1787870236.613789 write 0x003d95 thru 0x0101bc (0xc428 bytes) 000856 1787870236.641014 read 0x0223d0 thru 0x02c23b (0x9e6c bytes) 000857 1787870236.658027 mapread 0x0263c6 thru 0x03044e (0xa089 bytes) 000860 1787870236.715454 read 0x0350cc thru 0x03b462 (0x6397 bytes) 000861 1787870236.716215 trunc from 0x03ce39 to 0x001196 (0x3bca2 bytes) 000862 1787870236.762480 mapread 0x000907 thru 0x001196 (0x890 bytes) 000863 1787870236.773340 mapwrite 0x01af75 thru 0x021be1 (0x6c6d bytes) 000865 1787870236.879589 write 0x03af72 thru 0x03ffff (0x508e bytes) 000866 1787870236.904653 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xbe2711d3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbac0d1b5 Mirror 17: 0xbac0d1b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000867 1787870238.650183 fallocate 0x01262a thru 0x0131e0 (0xbb7 bytes) 000868 1787870238.713523 read 0x00e7ba thru 0x00fed4 (0x171b bytes) 000869 1787870238.734655 mapwrite 0x006d55 thru 0x015ace (0xed7a bytes) 000870 1787870238.863928 trunc from 0x021a15 to 0x027104 (0x56f0 bytes) 000871 1787870238.874619 trunc from 0x040000 to 0x001746 (0x3e8b9 bytes) 000872 1787870238.950919 mapread 0x000d0a thru 0x001746 (0xa3d bytes) 000873 1787870238.968272 fallocate 0x00d588 thru 0x01cfed (0xfa66 bytes) 000874 1787870238.977614 write 0x017972 thru 0x01c37a (0x4a09 bytes) 000876 1787870238.981647 punch from 0x00dc0a to 0x00ebd6 (0xfcd bytes) 000877 1787870238.994502 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c37b): Mirror 17: 0xc20a1103 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c37b): Mirror 17: 0xc20a1103 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c37b: skipped 000879 1787870239.645043 read 0x01679e thru 0x01b842 (0x50a5 bytes) 000880 1787870239.671842 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000881 1787870239.952839 trunc from 0x01c37b to 0x02a6a1 (0xe327 bytes) 000882 1787870239.979666 punch from 0x000d86 to 0x00d3aa (0xc625 bytes) 000883 1787870239.985777 write 0x020e6f thru 0x021baa (0xd3c bytes) 000884 1787870239.996722 trunc from 0x01b4c3 to 0x025fd0 (0xab0e bytes) 000885 1787870240.016792 punch from 0x017616 to 0x01f495 (0x7e80 bytes) 000886 1787870240.026277 mapwrite 0x03146f thru 0x034eb1 (0x3a43 bytes) 000887 1787870240.142766 write 0x00747e thru 0x0115c4 (0xa147 bytes) 000888 1787870240.159165 mapwrite 0x0334c2 thru 0x0390b8 (0x5bf7 bytes) 000889 1787870240.219975 mapread 0x00487a thru 0x010b14 (0xc29b bytes) 000891 1787870240.246613 fallocate 0x034c32 thru 0x03e405 (0x97d4 bytes) 000892 1787870240.253312 mapwrite 0x0367f4 thru 0x03ce54 (0x6661 bytes) 000894 1787870240.296800 MIRROR_EXTEND to 2 mirrors 000895 1787870241.341380 read 0x0041ab thru 0x00e13c (0x9f92 bytes) 000896 1787870241.357069 write 0x000328 thru 0x003930 (0x3609 bytes) 000897 1787870241.508718 mapwrite 0x033f20 thru 0x038990 (0x4a71 bytes) 000898 1787870241.595621 trunc from 0x02771a to 0x0298ec (0x21d3 bytes) 000899 1787870241.603805 read 0x031265 thru 0x034a98 (0x3834 bytes) 000900 1787870241.605344 fallocate 0x035d0a thru 0x03eb45 (0x8e3c bytes) 000901 1787870241.614564 trunc from 0x00b20f to 0x015dd6 (0xabc8 bytes) 000902 1787870241.625382 mapwrite 0x028946 thru 0x0302ad (0x7968 bytes) 000904 1787870241.711935 punch from 0x003946 to 0x012abd (0xf178 bytes) 000905 1787870241.721306 fallocate 0x0131c6 thru 0x021593 (0xe3ce bytes) 000906 1787870241.728492 punch from 0x0189bc to 0x0241d5 (0xb81a bytes) 000907 1787870241.739932 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000908 1787870241.929401 write 0x01366b thru 0x01d0dc (0x9a72 bytes) 000909 1787870241.963574 MIRROR_EXTEND to 2 mirrors 000910 1787870242.951786 mapread 0x0226cf thru 0x02ff22 (0xd854 bytes) 000911 1787870242.996477 trunc from 0x01f9db to 0x025b1a (0x6140 bytes) 000912 1787870243.061385 trunc from 0x02f848 to 0x034f96 (0x574f bytes) 000913 1787870243.067090 trunc from 0x00caa0 to 0x012cf5 (0x6256 bytes) 000914 1787870243.074484 fallocate 0x02a206 thru 0x0381a9 (0xdfa4 bytes) 000915 1787870243.081305 punch from 0x01109d to 0x020457 (0xf3bb bytes) 000918 1787870243.088055 mapread 0x039214 thru 0x03dc35 (0x4a22 bytes) 000919 1787870243.101431 trunc from 0x035870 to 0x036df2 (0x1583 bytes) 000920 1787870243.113328 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000921 1787870243.271316 write 0x007c0a thru 0x00a7c0 (0x2bb7 bytes) 000922 1787870243.301106 MIRROR_EXTEND to 2 mirrors 000923 1787870244.334843 punch from 0x033bca to 0x038549 (0x4980 bytes) 000924 1787870244.406914 read 0x03e2c1 thru 0x03e405 (0x145 bytes) 000925 1787870244.424677 mapwrite 0x0304d4 thru 0x036d80 (0x68ad bytes) 000927 1787870244.520369 fallocate 0x03534f thru 0x03ffff (0xacb1 bytes) 000928 1787870244.533280 trunc from 0x019a57 to 0x021b86 (0x8130 bytes) 000929 1787870244.541070 read 0x02b5e3 thru 0x034a9e (0x94bc bytes) 000931 1787870244.542414 read 0x036859 thru 0x03e405 (0x7bad bytes) 000932 1787870244.568480 mapread 0x03d6e7 thru 0x03e405 (0xd1f bytes) 000933 1787870244.569359 mapread 0x0204f4 thru 0x02adf7 (0xa904 bytes) 000934 1787870244.570260 mapwrite 0x006f34 thru 0x00aa9a (0x3b67 bytes) 000935 1787870244.592072 punch from 0x03b9cb to 0x03e405 (0x2a3b bytes) 000936 1787870244.600086 trunc from 0x03e406 to 0x030847 (0xdbbe bytes) 000937 1787870244.636874 punch from 0x001d9c to 0x00986e (0x7ad3 bytes) 000938 1787870244.644556 trunc from 0x0296ae to 0x02982d (0x180 bytes) 000939 1787870244.649218 punch from 0x020e07 to 0x020ee8 (0xe2 bytes) 000940 1787870244.660127 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000942 1787870244.829101 trunc from 0x030848 to 0x001fa7 (0x2e8a0 bytes) 000943 1787870244.873462 trunc from 0x0017ac to 0x001fa7 (0x7fc bytes) 000945 1787870244.879930 mapread 0x0011e9 thru 0x001fa7 (0xdbf bytes) 000946 1787870244.890574 mapwrite 0x0088cc thru 0x0169c0 (0xe0f5 bytes) 000947 1787870244.954547 mapread 0x016443 thru 0x0169c0 (0x57e bytes) 000948 1787870244.955680 mapread 0x00ea40 thru 0x0169c0 (0x7f81 bytes) 000949 1787870244.956628 mapread 0x00e3bd thru 0x011e4a (0x3a8e bytes) 000950 1787870244.957751 trunc from 0x0169c1 to 0x037110 (0x20750 bytes) 000951 1787870244.984598 write 0x034d6f thru 0x03c41b (0x76ad bytes) 000953 1787870245.005638 mapread 0x01a30c thru 0x023c64 (0x9959 bytes) 000954 1787870245.041768 mapwrite 0x03c2c8 thru 0x03ffff (0x3d38 bytes) 000955 1787870245.116709 fallocate 0x0013a4 thru 0x010471 (0xf0ce bytes) 000956 1787870245.124403 fallocate 0x02c9a7 thru 0x0373b4 (0xaa0e bytes) 000957 1787870245.132140 read 0x01c1da thru 0x0269c8 (0xa7ef bytes) 000958 1787870245.132746 trunc from 0x040000 to 0x038e0b (0x71f4 bytes) 000960 1787870245.169367 mapread 0x01b06f thru 0x021477 (0x6409 bytes) 000961 1787870245.172166 trunc from 0x038e0c to 0x037579 (0x1892 bytes) 000962 1787870245.200525 write 0x0044b1 thru 0x01320d (0xed5d bytes) 000963 1787870245.204199 trunc from 0x03001c to 0x037579 (0x755e bytes) 000964 1787870245.217345 trunc from 0x036f39 to 0x037579 (0x641 bytes) 000965 1787870245.230279 read 0x008a7e thru 0x016f0f (0xe492 bytes) 000966 1787870245.230900 read 0x022b9f thru 0x0263b6 (0x3818 bytes) 000967 1787870245.233199 read 0x033884 thru 0x037579 (0x3cf6 bytes) 000968 1787870245.251219 read 0x034a38 thru 0x037579 (0x2b42 bytes) 000971 1787870245.262744 MIRROR_EXTEND to 2 mirrors 000972 1787870246.005207 read 0x00af28 thru 0x010f98 (0x6071 bytes) 000973 1787870246.017082 fallocate 0x0201e0 thru 0x02feda (0xfcfb bytes) 000974 1787870246.053725 write 0x024778 thru 0x031577 (0xce00 bytes) 000975 1787870246.084732 mapwrite 0x0384e5 thru 0x03e93c (0x6458 bytes) 000976 1787870246.189135 fallocate 0x02724f thru 0x0309ed (0x979f bytes) 000977 1787870246.203506 mapwrite 0x02abd2 thru 0x03289b (0x7cca bytes) 000978 1787870246.265538 trunc from 0x03e93d to 0x03d5ea (0x1352 bytes) 000979 1787870246.286538 write 0x02ec80 thru 0x03d964 (0xece5 bytes) 000980 1787870246.292489 punch from 0x00e0b8 to 0x010bbe (0x2b07 bytes) 000981 1787870246.300053 mapread 0x038ce2 thru 0x03ccfd (0x401c bytes) 000982 1787870246.306216 MIRROR_RESYNC 2 mirrors 000984 1787870247.336580 fallocate 0x0374d9 thru 0x03ffff (0x8b27 bytes) 000985 1787870247.386502 mapwrite 0x03a32d thru 0x03ba37 (0x170b bytes) 000986 1787870247.414926 mapread 0x01151e thru 0x01689b (0x537e bytes) 000987 1787870247.429211 fallocate 0x01b7b5 thru 0x027dcc (0xc618 bytes) 000989 1787870247.436011 mapwrite 0x023a0a thru 0x02f5e0 (0xbbd7 bytes) 000990 1787870247.475224 mapwrite 0x031a60 thru 0x03dbc6 (0xc167 bytes) 000991 1787870247.511035 mapwrite 0x004beb thru 0x00cf36 (0x834c bytes) 000992 1787870247.550526 trunc from 0x040000 to 0x03b267 (0x4d98 bytes) 000993 1787870247.612143 trunc from 0x024d93 to 0x02e704 (0x9972 bytes) 000994 1787870247.622979 read 0x0202f5 thru 0x02cf06 (0xcc12 bytes) 000995 1787870247.634270 trunc from 0x022441 to 0x02a697 (0x8257 bytes) 000996 1787870247.639224 punch from 0x007721 to 0x0164ab (0xed8b bytes) 000997 1787870247.646169 mapread 0x016d46 thru 0x01d2d5 (0x6590 bytes) 000998 1787870247.661441 MIRROR_RESYNC 2 mirrors 000999 1787870248.443182 write 0x02af8a thru 0x032b60 (0x7bd7 bytes) 001000 1787870248.506833 MIRROR_RESYNC 2 mirrors All operations completed A-OK! PASS 70a (119s) == sanity-flr test 71: check mirror extend parent fid ==== 18:37:36 (1787870256) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x141:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (14s) == sanity-flr test 72: check append on FLR file ========== 18:37:51 (1787870271) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.293024 s, 7.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216476 s, 4.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x137:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 18:38:02 (1787870282) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7225344 .. Extending file size to 5597376 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9120448 .. Extending file size to 8968224 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62422 62423 62424 62427 62428 done 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) 192.168.204.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (86s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:39:29 (1787870369) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7272608 .. Extending file size to 2077472 .. failed Extending file size to 7852352 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8684544 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7697632 .. Extending file size to 9385888 .. Extending file size to 8761024 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8817664 .. Waiting 64145 64147 64148 64150 64152 failed 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) 192.168.204.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:40:50 (1787870450) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1268916 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25112 3581908 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19116 3587904 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44228 7169812 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2388 1268916 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25112 3581908 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 19116 3587904 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44228 7169812 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139667 s, 15.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195254 s, 10.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x3353a931 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:41:13 (1787870473) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14039 s, 14.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 18:41:21 (1787870481) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x154:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x154:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:41:29 (1787870489) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123125 s, 17.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1aa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ab:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:41:37 (1787870497) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126777 s, 16.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1a9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ac:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ad:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:41:47 (1787870507) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154409 s, 13.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1aa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ae:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1af:0x0] } PASS 204c (38s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:42:25 (1787870545) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140282 s, 7.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ab:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ab:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1b0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ac:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:43:24 (1787870604) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182908 s, 11.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x156:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x156:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:44:19 (1787870659) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg424-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg424-server: Pool lustre.test_204f created Adding targets to pool oleg424-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg424-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106051 s, 19.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x157:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x157:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg424-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg424-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg424-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 18:45:37 (1787870737) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 18:45:47 (1787870747) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 18:45:56 (1787870756) oleg424-server: Pool lustre.test_206 created Adding targets to pool oleg424-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg424-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg424-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg424-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg424-server: Pool lustre.test_206 destroyed PASS 206 (32s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 18:46:28 (1787870788) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753877 s, 13.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903423 s, 11.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2144c0f5 Mirror 3: 0x2144c0f5 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 18:46:40 (1787870800) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 18:46:45 (1787870805) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 18:46:49 (1787870809) Starting client: oleg424-client.virtnet: -o user_xattr,flock 192.168.204.124@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0568141 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2460000 Read this (4 bytes): BBBB 192.168.204.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg424-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 18:47:31 (1787870851) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0170673 s, 480 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 18:47:41 (1787870861) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540528 s, 19.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbd lmm_fid: [0x200000405:0xbd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbd lmm_fid: [0x200000405:0xbd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x161:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611325 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 18:47:43.000000000 -0400 Modify: 2026-08-27 18:47:43.000000000 -0400 Change: 2026-08-27 18:47:45.000000000 -0400 Birth: 2026-08-27 18:47:43.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 18:47:51 (1787870871) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540449 s, 19.4 MB/s oleg424-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg424-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg424-server: O/280000401/d0/160: File not found by ext2_lookup PASS 210b (27s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 18:48:18 (1787870898) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.338625 s, 121 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0254598 s, 161 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 18:48 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc2:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 18:48 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 18:48:30 (1787870910) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 18:48:38 (1787870918) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 2099 sec =========== 18:48:48 (1787870928) === sanity-flr: start cleanup 18:48:50 (1787870930) === === sanity-flr: finish cleanup 18:49:01 (1787870941) ===