-----============= acceptance-small: sanity-flr ============----- Fri Aug 28 01:00:46 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg623-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:01:06 (1787893266) === oleg623-client.virtnet: executing check_config_client /mnt/lustre oleg623-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg623-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fc251ec2800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fc251ec2800.idle_timeout=debug disable quota as required oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:01:48 (1787893308) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8238] == sanity-flr test 0a: lfs mirror create with -N option == 01:01:51 (1787893311) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:02:08 (1787893328) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:02:11 (1787893331) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:02:15 (1787893335) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:02:40 (1787893360) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:02:43 (1787893363) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:02:47 (1787893367) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9fc251ec2800.stats=clear osc.lustre-OST0001-osc-ffff9fc251ec2800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:02:54 (1787893374) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:03:04 (1787893384) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:03:11 (1787893391) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:03:14 (1787893394) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:03:22 (1787893402) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 01:03:31 (1787893411) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:03:37 (1787893417) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:03:44 (1787893424) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:03:51 (1787893431) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:04:00 (1787893440) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:04:05 (1787893445) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.11586 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537467 s, 19.5 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:04:13 (1787893453) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0814768 s, 38.6 MB/s PASS 22 (6s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:04:20 (1787893460) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:04:29 (1787893469) Stopping /mnt/lustre-ost1 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fc251ec2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fc251ec2800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg623-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg623-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fc251ec2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fc251ec2800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg623-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (134s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:06:42 (1787893602) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.566443 s, 16.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fc251ec2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fc251ec2800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg623-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:07:22 (1787893642) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fc251ec2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fc251ec2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg623-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fc251ec2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fc251ec2800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg623-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:08:44 (1787893724) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.90853 s, 18.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 28 01:08 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.903s user 0m0.000s sys 0m0.199s Stopping /mnt/lustre-ost1 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fc251ec2800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fc251ec2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.095s user 0m0.000s sys 0m0.194s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg623-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg623-server oleg623-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fc251ec2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fc251ec2800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.075s user 0m0.000s sys 0m0.162s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg623-server: oleg623-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg623-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg623-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fc251ec2800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fc251ec2800.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:10:13 (1787893813) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:10:17 (1787893817) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:10:21 (1787893821) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 01:10:25 (1787893825) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 01:10:34 (1787893834) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.496825 s, 25.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:10:47 (1787893847) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418498 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244711 s, 42.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:10:54 (1787893854) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:11:02 (1787893862) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00178086 s, 575 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00211932 s, 483 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00176648 s, 580 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00138827 s, 738 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 01:11:11 (1787893871) create test files with size 15738753 .. 3eac6550ab52b8e491961d19aa3d53ff - d48fa088141930e7bdf25ec3cb2afaec - d01359d014ba05561404a9c54850860b - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg623-client.virtnet /mnt/lustre (opts:) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 01:11:31 (1787893891) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:11:45 (1787893905) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:11:51 (1787893911) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759683 s, 13.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554063 s, 18.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 01:12:01 (1787893921) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138567 s, 15.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.326677 s, 12.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.267644 s, 7.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 01:12:19 (1787893939) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:12:22 (1787893942) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:12:26 (1787893946) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466487 s, 22.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 01:12:33 (1787893953) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.385198 s, 8.2 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:12:44 (1787893964) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:12:53 (1787893973) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.350893 s, 29.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg623-client.virtnet /mnt/lustre (opts:) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:13:04 (1787893984) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9fc260003000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.374326 s, 28.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg623-client.virtnet /mnt/lustre (opts:) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9fc24307d800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:13:22 (1787894002) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:13:58 (1787894038) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:14:31 (1787894071) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 01:14:34 (1787894074) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.272328 s, 11.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.118944 s, 26.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8bf05ef9 (nosync,stale) Mirror 2: 0x8bf05ef9 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 37a15db8b674a8f810e7603611232720 - mirror 1 checksum: 856ca01ba2aa1a0acfab5a7cda96cf51 - mirror 2 checksum: 37a15db8b674a8f810e7603611232720 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-28T01:14:38-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-28T01:14:38-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 37a15db8b674a8f810e7603611232720 - mirror 1 checksum: 856ca01ba2aa1a0acfab5a7cda96cf51 - mirror 2 checksum: 856ca01ba2aa1a0acfab5a7cda96cf51 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-28T01:14:38-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-28T01:14:38-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:14:45 (1787894085) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264716 s, 39.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:14:51 (1787894091) Failing mds1 on oleg623-server Stopping /mnt/lustre-mds1 (opts:) on oleg623-server 01:15:04 (1787894104) shut down facet: mds1 facet_host: oleg623-server facet_failover_host: oleg623-server Failover mds1 to oleg623-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg623-server: oleg623-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-MDT0000 01:15:35 (1787894135) targets are mounted 01:15:35 (1787894135) facet_failover done oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248338 s, 42.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } PASS 50A (62s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:15:53 (1787894153) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0574057 s, 357 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.020093 s, 1.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:16:04 (1787894164) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0744183 s, 275 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0211825 s, 967 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:16:14 (1787894174) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0130408 s, 1.3 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:16:22 (1787894182) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0702517 s, 292 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (11s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:16:33 (1787894193) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00805305 s, 509 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0502328 s, 81.5 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:16:41 (1787894201) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:17:03 (1787894223) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:17:31 (1787894251) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:17:53 (1787894273) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:18:15 (1787894295) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 01:18:22 (1787894302) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57614 mirror split pid 57615 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57614 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57615 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 01:19:30 (1787894370) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 4372 000001 1787894372.478746 write 0x006688 thru 0x015939 (0xf2b2 bytes) 000003 1787894372.496952 write 0x009e96 thru 0x00ac6a (0xdd5 bytes) fallocating to largest ever: 0x287c6 000004 1787894372.499272 fallocate 0x01a3a2 thru 0x0287c5 (0xe424 bytes) 000006 1787894372.505955 trunc from 0x0287c6 to 0x01d8d6 (0xaeef bytes) 000007 1787894372.527866 write 0x015e25 thru 0x022784 (0xc960 bytes) 000008 1787894372.544995 MIRROR_EXTEND to 2 mirrors 000009 1787894373.466870 write 0x00c940 thru 0x0122ed (0x59ae bytes) fallocating to largest ever: 0x3572b 000010 1787894373.523254 fallocate 0x02c664 thru 0x03572a (0x90c7 bytes) 000011 1787894373.528707 mapwrite 0x019714 thru 0x01eb3e (0x542b bytes) 000012 1787894373.575154 mapwrite 0x00f6f7 thru 0x01aeb0 (0xb7ba bytes) 000013 1787894373.604879 read 0x01f234 thru 0x022784 (0x3551 bytes) 000014 1787894373.605561 write 0x0313ed thru 0x034c26 (0x383a bytes) 000015 1787894373.607756 punch from 0x00f8c2 to 0x01ca03 (0xd142 bytes) 000016 1787894373.613304 punch from 0x0337ec to 0x034c26 (0x143b bytes) 000017 1787894373.640462 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34c27): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34c27): Mirror 2: 0x88c51f25 (stale) CRC-32 checksum value for chunk [0, 0x34c27): Mirror 1: 0x747d0b70 Mirror 3: 0x747d0b70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34c27: skipped 000018 1787894374.901974 write 0x000753 thru 0x0038e5 (0x3193 bytes) fallocating to largest ever: 0x3fa1b 000019 1787894374.946950 fallocate 0x038e0e thru 0x03fa1a (0x6c0d bytes) 000020 1787894374.952262 trunc from 0x03fa1b to 0x022482 (0x1d598 bytes) 000021 1787894374.975764 punch from 0x007c4e to 0x016893 (0xec46 bytes) 000022 1787894374.983239 mapread 0x009a19 thru 0x012df4 (0x93dc bytes) 000023 1787894375.006749 read 0x00b6f6 thru 0x01064c (0x4f57 bytes) 000024 1787894375.007283 read 0x006490 thru 0x009867 (0x33d8 bytes) 000025 1787894375.007801 trunc from 0x019f8c to 0x022482 (0x84f7 bytes) 000028 1787894375.036115 mapwrite 0x00d3d9 thru 0x0141ec (0x6e14 bytes) 000029 1787894375.081071 trunc from 0x021470 to 0x022482 (0x1013 bytes) 000030 1787894375.087572 trunc from 0x01bed6 to 0x022482 (0x65ad bytes) 000032 1787894375.100401 MIRROR_RESYNC 3 mirrors 000033 1787894376.217529 mapwrite 0x0347f4 thru 0x036a80 (0x228d bytes) 000034 1787894376.385548 fallocate 0x02aaee thru 0x030e08 (0x631b bytes) 000035 1787894376.391443 mapread 0x0007b1 thru 0x0093b6 (0x8c06 bytes) 000036 1787894376.400561 trunc from 0x036a81 to 0x012267 (0x24819 bytes) 000037 1787894376.419122 mapread 0x010b31 thru 0x012267 (0x1737 bytes) 000039 1787894376.429411 punch from 0x00a6be to 0x012267 (0x7baa bytes) 000040 1787894376.435322 write 0x01d6bc thru 0x01f976 (0x22bb bytes) 000041 1787894376.442190 MIRROR_RESYNC 3 mirrors 000043 1787894377.103034 fallocate 0x00242a thru 0x0112e4 (0xeebb bytes) 000044 1787894377.163174 write 0x0371e6 thru 0x03ffff (0x8e1a bytes) 000045 1787894377.172854 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9944496e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9944496e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000046 1787894377.615325 trunc from 0x040000 to 0x01139e (0x2ec61 bytes) 000047 1787894377.699238 trunc from 0x01139f to 0x020b55 (0xf7b7 bytes) 000048 1787894377.719390 trunc from 0x005e2c to 0x01348a (0xd65f bytes) 000049 1787894377.725563 punch from 0x00d0c1 to 0x01cb72 (0xfab2 bytes) 000050 1787894377.730292 trunc from 0x01dc62 to 0x020b55 (0x2ef4 bytes) 000051 1787894377.736112 mapread 0x01b0cb thru 0x020b55 (0x5a8b bytes) 000052 1787894377.752114 trunc from 0x020b56 to 0x031b11 (0x10fbc bytes) 000054 1787894377.763413 mapwrite 0x0186db thru 0x01fdab (0x76d1 bytes) 000055 1787894377.803955 MIRROR_RESYNC 2 mirrors 000056 1787894378.412735 punch from 0x00f154 to 0x01dd54 (0xec01 bytes) 000057 1787894378.444527 write 0x02ba5c thru 0x030104 (0x46a9 bytes) 000058 1787894378.452761 fallocate 0x02c529 thru 0x035519 (0x8ff1 bytes) 000059 1787894378.459444 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31b12): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b12): Mirror 2: 0x81253c65 (stale) CRC-32 checksum value for chunk [0, 0x31b12): Mirror 1: 0x954e8f23 Mirror 3: 0x954e8f23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b12: skipped 000061 1787894379.392799 read 0x00c4cf thru 0x00cd6f (0x8a1 bytes) 000062 1787894379.403841 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31b12): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b12): Mirror 2: 0x81253c65 (stale) CRC-32 checksum value for chunk [0, 0x31b12): Mirror 1: 0x954e8f23 Mirror 3: 0x954e8f23 Mirror 4: 0x954e8f23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b12: skipped 000063 1787894380.526860 fallocate 0x0048a6 thru 0x011723 (0xce7e bytes) 000064 1787894380.574195 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31b12): Mirror 3: 0x954e8f23 (stale) Mirror 4: 0x954e8f23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b12): Mirror 3: 0x954e8f23 (stale) Mirror 4: 0x954e8f23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b12: skipped 000066 1787894380.923970 trunc from 0x009b91 to 0x01520c (0xb67c bytes) 000067 1787894380.933807 mapwrite 0x002629 thru 0x003c81 (0x1659 bytes) 000068 1787894380.983720 mapwrite 0x01e935 thru 0x022295 (0x3961 bytes) 000070 1787894381.043262 read 0x02347b thru 0x02ad9f (0x7925 bytes) 000071 1787894381.052668 trunc from 0x031b12 to 0x00c5fd (0x25514 bytes) 000072 1787894381.081797 fallocate 0x02b8a0 thru 0x02d576 (0x1cd7 bytes) 000073 1787894381.094068 mapread 0x005ac9 thru 0x00c5fd (0x6b35 bytes) 000074 1787894381.119112 MIRROR_RESYNC 3 mirrors 000075 1787894382.104165 trunc from 0x00b3ab to 0x00c5fd (0x1253 bytes) 000077 1787894382.177373 MIRROR_RESYNC 3 mirrors 000078 1787894382.856420 punch from 0x006713 to 0x00c5fd (0x5eeb bytes) 000079 1787894382.881157 trunc from 0x00c1b5 to 0x00c5fd (0x449 bytes) 000080 1787894382.886718 fallocate 0x00ccb6 thru 0x019b56 (0xcea1 bytes) 000081 1787894382.894599 MIRROR_RESYNC 3 mirrors 000082 1787894383.533477 trunc from 0x019b57 to 0x0167a9 (0x33ad bytes) 000083 1787894383.570974 trunc from 0x0167aa to 0x0284d9 (0x11d30 bytes) 000084 1787894383.591548 trunc from 0x0284da to 0x03d23b (0x14d62 bytes) 000085 1787894383.605097 fallocate 0x01c098 thru 0x025dcd (0x9d36 bytes) 000086 1787894383.609589 mapread 0x0257cf thru 0x031e49 (0xc67b bytes) 000087 1787894383.631768 trunc from 0x03d23c to 0x019024 (0x24217 bytes) 000088 1787894383.654565 write 0x002ba5 thru 0x00c45b (0x98b7 bytes) 000089 1787894383.665393 read 0x00ab5f thru 0x016a15 (0xbeb7 bytes) 000090 1787894383.672937 mapread 0x004df0 thru 0x00b8a4 (0x6ab5 bytes) 000091 1787894383.674202 trunc from 0x018f53 to 0x019024 (0xd2 bytes) 000092 1787894383.679726 fallocate 0x018111 thru 0x020dda (0x8cca bytes) 000093 1787894383.684493 trunc from 0x020ddb to 0x00b1d1 (0x15c09 bytes) 000094 1787894383.700022 trunc from 0x008966 to 0x00b1d1 (0x286c bytes) 000095 1787894383.714854 trunc from 0x004504 to 0x0062c4 (0x1dc1 bytes) 000097 1787894383.720974 trunc from 0x00b1d2 to 0x005734 (0x5a9d bytes) 000099 1787894383.746189 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x5735): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5735): Mirror 3: 0x68aa9ca5 (stale) Mirror 4: 0x68aa9ca5 (stale) CRC-32 checksum value for chunk [0, 0x5735): Mirror 1: 0x71e70697 Mirror 5: 0x71e70697 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5735: skipped 000100 1787894384.470495 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x5735): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5735): Mirror 3: 0x68aa9ca5 (stale) Mirror 4: 0x68aa9ca5 (stale) CRC-32 checksum value for chunk [0, 0x5735): Mirror 1: 0x71e70697 Mirror 5: 0x71e70697 Mirror 6: 0x71e70697 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5735: skipped 000101 1787894385.204137 punch from 0x002333 to 0x005734 (0x3402 bytes) 000103 1787894385.247423 write 0x039b5e thru 0x03ffff (0x64a2 bytes) 000104 1787894385.256418 punch from 0x0106bd to 0x01e5a4 (0xdee8 bytes) 000105 1787894385.267473 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8bd8688 (stale) Mirror 4: 0x8bd8688 (stale) Mirror 5: 0xfaf60634 (stale) Mirror 6: 0xfaf60634 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x51a785fd Mirror 7: 0x51a785fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000106 1787894386.415903 MIRROR_RESYNC 6 mirrors 000107 1787894387.757480 trunc from 0x040000 to 0x014c40 (0x2b3bf bytes) 000108 1787894387.812711 write 0x00a634 thru 0x00f413 (0x4de0 bytes) 000109 1787894387.831513 MIRROR_RESYNC 6 mirrors 000110 1787894389.102219 punch from 0x00dc23 to 0x010ff9 (0x33d7 bytes) 000111 1787894389.131263 trunc from 0x014c41 to 0x03d774 (0x28b34 bytes) 000112 1787894389.162365 mapwrite 0x0100c7 thru 0x01958c (0x94c6 bytes) 000113 1787894389.222319 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d775): Mirror 4: 0x3136e677 (stale) Mirror 5: 0x3136e677 (stale) Mirror 6: 0x3136e677 (stale) Mirror 7: 0x3136e677 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d775): Mirror 4: 0x3136e677 (stale) Mirror 5: 0x3136e677 (stale) Mirror 6: 0x3136e677 (stale) Mirror 7: 0x3136e677 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d775: skipped 000114 1787894389.643256 trunc from 0x029974 to 0x02f1dd (0x586a bytes) 000115 1787894389.651241 fallocate 0x01f2b2 thru 0x025705 (0x6454 bytes) 000116 1787894389.658435 read 0x01d95b thru 0x02921f (0xb8c5 bytes) 000118 1787894389.673175 fallocate 0x0215f0 thru 0x02aee0 (0x98f1 bytes) 000119 1787894389.683868 read 0x02b9bb thru 0x02ea1b (0x3061 bytes) 000120 1787894389.697987 write 0x03b142 thru 0x03ffff (0x4ebe bytes) 000121 1787894389.705179 punch from 0x022bd2 to 0x032a85 (0xfeb4 bytes) 000122 1787894389.745887 read 0x003c47 thru 0x0128f9 (0xecb3 bytes) 000124 1787894389.747566 trunc from 0x040000 to 0x00061a (0x3f9e5 bytes) 000125 1787894389.772083 write 0x005628 thru 0x013034 (0xda0d bytes) 000126 1787894389.773735 mapwrite 0x026dd2 thru 0x031a5d (0xac8c bytes) 000127 1787894389.832005 read 0x00a89e thru 0x01955a (0xecbd bytes) 000128 1787894389.833688 read 0x024126 thru 0x0281a4 (0x407f bytes) 000129 1787894389.834274 mapwrite 0x025847 thru 0x034317 (0xead1 bytes) 000130 1787894389.892825 trunc from 0x034318 to 0x010658 (0x23cbf bytes) 000131 1787894389.918970 read 0x00b7b0 thru 0x010658 (0x4ea9 bytes) 000132 1787894389.919741 trunc from 0x010659 to 0x032824 (0x221cc bytes) 000133 1787894389.947502 read 0x008cc2 thru 0x00f47d (0x67bc bytes) 000134 1787894389.948398 punch from 0x00dec9 to 0x015388 (0x74c0 bytes) 000135 1787894389.965426 mapwrite 0x0109df thru 0x0123e4 (0x1a06 bytes) 000136 1787894390.012225 fallocate 0x00b08e thru 0x00cb27 (0x1a9a bytes) 000137 1787894390.016347 fallocate 0x00b321 thru 0x01426b (0x8f4b bytes) 000138 1787894390.023710 fallocate 0x035ca9 thru 0x037c51 (0x1fa9 bytes) 000140 1787894390.030563 trunc from 0x001818 to 0x0097f3 (0x7fdc bytes) 000141 1787894390.036361 write 0x013e5d thru 0x0229b9 (0xeb5d bytes) 000142 1787894390.038233 punch from 0x02cd7b to 0x032824 (0x5aaa bytes) 000143 1787894390.044182 trunc from 0x00a4a9 to 0x016e9b (0xc9f3 bytes) 000144 1787894390.064375 read 0x02175b thru 0x02e468 (0xcd0e bytes) 000145 1787894390.090148 punch from 0x01c32d to 0x0264e8 (0xa1bc bytes) 000146 1787894390.095144 fallocate 0x034109 thru 0x03b8a7 (0x779f bytes) 000147 1787894390.101112 read 0x0157e1 thru 0x01ba4f (0x626f bytes) 000148 1787894390.103592 write 0x03ce87 thru 0x03ffff (0x3179 bytes) 000149 1787894390.105059 punch from 0x0268a9 to 0x02aa8e (0x41e6 bytes) 000151 1787894390.116364 MIRROR_RESYNC 5 mirrors 000152 1787894391.603539 mapread 0x03c9b6 thru 0x03ffff (0x364a bytes) 000153 1787894391.642796 fallocate 0x001830 thru 0x00994a (0x811b bytes) 000155 1787894391.685819 mapread 0x00d2d6 thru 0x014bf6 (0x7921 bytes) 000156 1787894391.709551 punch from 0x001db0 to 0x00d6d3 (0xb924 bytes) 000157 1787894391.720396 mapwrite 0x016cd8 thru 0x01d1eb (0x6514 bytes) 000158 1787894391.768220 trunc from 0x040000 to 0x036faa (0x9055 bytes) 000159 1787894391.795826 punch from 0x00d292 to 0x017325 (0xa094 bytes) 000160 1787894391.807996 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36fab): Mirror 4: 0xedc8990b (stale) Mirror 5: 0xedc8990b (stale) Mirror 6: 0xedc8990b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36fab): Mirror 4: 0xedc8990b (stale) Mirror 5: 0xedc8990b (stale) Mirror 6: 0xedc8990b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36fab: skipped 000161 1787894392.204497 trunc from 0x012359 to 0x01a222 (0x7eca bytes) 000162 1787894392.215201 read 0x00104c thru 0x0020a1 (0x1056 bytes) 000163 1787894392.222196 read 0x02e20e thru 0x036faa (0x8d9d bytes) 000164 1787894392.234158 trunc from 0x036fab to 0x0006cb (0x368df bytes) 000165 1787894392.259063 mapwrite 0x006237 thru 0x00cd0d (0x6ad7 bytes) 000166 1787894392.403162 fallocate 0x030130 thru 0x03c097 (0xbf68 bytes) 000167 1787894392.408716 trunc from 0x0300fe to 0x038ded (0x8cf0 bytes) 000169 1787894392.416923 mapread 0x017ac9 thru 0x01bc34 (0x416c bytes) 000170 1787894392.449474 mapwrite 0x03257f thru 0x03ffff (0xda81 bytes) 000171 1787894392.563655 mapwrite 0x0342ec thru 0x034db0 (0xac5 bytes) 000172 1787894392.574183 trunc from 0x01cb25 to 0x02388b (0x6d67 bytes) 000173 1787894392.579514 trunc from 0x040000 to 0x01ae53 (0x251ac bytes) 000174 1787894392.600093 trunc from 0x01ae54 to 0x0013ee (0x19a65 bytes) 000175 1787894392.615115 mapwrite 0x033150 thru 0x038fce (0x5e7f bytes) fallocating to largest ever: 0x3faa1 000177 1787894392.669262 fallocate 0x0327aa thru 0x03faa0 (0xd2f7 bytes) 000178 1787894392.678050 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3faa1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3faa1): Mirror 4: 0xaba65dd5 (stale) Mirror 5: 0xaba65dd5 (stale) Mirror 6: 0xaba65dd5 (stale) CRC-32 checksum value for chunk [0, 0x3faa1): Mirror 1: 0xbcfa01e4 Mirror 7: 0xbcfa01e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3faa1: skipped 000179 1787894393.516100 trunc from 0x03faa1 to 0x01c8f2 (0x231ae bytes) 000180 1787894393.587986 trunc from 0x013ead to 0x01ab34 (0x6c88 bytes) 000182 1787894393.596955 mapwrite 0x028a5e thru 0x02eb81 (0x6124 bytes) 000183 1787894393.648527 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2eb82): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb82): Mirror 4: 0x65eff147 (stale) Mirror 5: 0x65eff147 (stale) Mirror 6: 0x65eff147 (stale) Mirror 7: 0x5975bea4 (stale) CRC-32 checksum value for chunk [0, 0x2eb82): Mirror 1: 0xd017209d Mirror 8: 0xd017209d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb82: skipped 000184 1787894394.559916 mapwrite 0x024cb6 thru 0x0326bd (0xda08 bytes) 000186 1787894394.749957 mapwrite 0x01dc1d thru 0x01ee93 (0x1277 bytes) 000188 1787894394.793030 mapwrite 0x013750 thru 0x01737e (0x3c2f bytes) 000189 1787894394.849266 mapread 0x027543 thru 0x02ae04 (0x38c2 bytes) 000190 1787894394.850170 fallocate 0x0112c2 thru 0x01d1c7 (0xbf06 bytes) 000192 1787894394.870233 fallocate 0x02cab5 thru 0x037ed2 (0xb41e bytes) 000193 1787894394.881752 trunc from 0x037ed3 to 0x03b811 (0x393f bytes) 000194 1787894394.933523 trunc from 0x007401 to 0x00f4df (0x80df bytes) 000195 1787894394.948223 MIRROR_RESYNC 6 mirrors 000196 1787894397.098397 punch from 0x037792 to 0x039f81 (0x27f0 bytes) 000197 1787894397.162576 trunc from 0x03b812 to 0x017694 (0x2417d bytes) 000198 1787894397.222137 trunc from 0x00624e to 0x00cb6b (0x691e bytes) 000199 1787894397.227951 mapwrite 0x016398 thru 0x01b9e1 (0x564a bytes) 000200 1787894397.286687 mapwrite 0x027c4a thru 0x032e09 (0xb1c0 bytes) 000201 1787894397.339738 trunc from 0x032e0a to 0x03d038 (0xa22f bytes) 000202 1787894397.356747 punch from 0x0083d7 to 0x00a79b (0x23c5 bytes) 000203 1787894397.362752 punch from 0x005ffd to 0x010ba6 (0xabaa bytes) 000204 1787894397.375350 trunc from 0x027a20 to 0x02c0ee (0x46cf bytes) 000205 1787894397.394902 trunc from 0x03d039 to 0x02e4f2 (0xeb46 bytes) 000206 1787894397.426189 trunc from 0x020054 to 0x029fe7 (0x9f94 bytes) 000207 1787894397.433034 read 0x02a9ba thru 0x02e4f2 (0x3b39 bytes) 000208 1787894397.439364 trunc from 0x02e4f3 to 0x02e4cb (0x27 bytes) 000209 1787894397.457424 mapwrite 0x0010b8 thru 0x00eb2a (0xda73 bytes) 000210 1787894397.514517 mapread 0x004a0a thru 0x006182 (0x1779 bytes) 000211 1787894397.520555 mapwrite 0x001624 thru 0x00731a (0x5cf7 bytes) 000212 1787894397.554181 fallocate 0x00dbe2 thru 0x0101d1 (0x25f0 bytes) 000213 1787894397.559665 mapwrite 0x0306c6 thru 0x03e55c (0xde97 bytes) 000214 1787894397.713959 punch from 0x026fa6 to 0x03374b (0xc7a6 bytes) 000215 1787894397.740329 mapread 0x01b3ee thru 0x01d60f (0x2222 bytes) 000216 1787894397.769103 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e55d): Mirror 4: 0x6d4ec061 (stale) Mirror 6: 0x6d4ec061 (stale) Mirror 7: 0x6d4ec061 (stale) Mirror 8: 0x6d4ec061 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e55d): Mirror 4: 0x6d4ec061 (stale) Mirror 6: 0x6d4ec061 (stale) Mirror 7: 0x6d4ec061 (stale) Mirror 8: 0x6d4ec061 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e55d: skipped 000217 1787894398.475983 read 0x025855 thru 0x0268cc (0x1078 bytes) 000218 1787894398.492351 trunc from 0x006bda to 0x00a2c5 (0x36ec bytes) 000219 1787894398.499005 read 0x00fc21 thru 0x01ef4d (0xf32d bytes) 000220 1787894398.578756 MIRROR_RESYNC 5 mirrors 000221 1787894400.239255 read 0x0240c4 thru 0x026ea3 (0x2de0 bytes) 000223 1787894400.245606 punch from 0x01b71a to 0x01f021 (0x3908 bytes) 000224 1787894400.284137 fallocate 0x0155ca thru 0x016a6e (0x14a5 bytes) 000225 1787894400.290136 mapread 0x03a0ff thru 0x03e55c (0x445e bytes) 000226 1787894400.307781 mapread 0x007aaf thru 0x007f50 (0x4a2 bytes) 000227 1787894400.322940 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e55d): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e55d): Mirror 4: 0x6720ac73 (stale) Mirror 6: 0x6720ac73 (stale) Mirror 7: 0x6720ac73 (stale) Mirror 8: 0x6720ac73 (stale) CRC-32 checksum value for chunk [0, 0x3e55d): Mirror 1: 0xf6d24901 Mirror 9: 0xf6d24901 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e55d: skipped 000228 1787894401.261257 fallocate 0x00a17c thru 0x01901c (0xeea1 bytes) 000229 1787894401.338687 read 0x0304bf thru 0x03e55c (0xe09e bytes) 000231 1787894401.354084 punch from 0x01434c to 0x016e15 (0x2aca bytes) 000232 1787894401.366893 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e55d): Mirror 4: 0x6720ac73 (stale) Mirror 6: 0x6720ac73 (stale) Mirror 7: 0x6720ac73 (stale) Mirror 8: 0x6720ac73 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e55d): Mirror 4: 0x6720ac73 (stale) Mirror 6: 0x6720ac73 (stale) Mirror 7: 0x6720ac73 (stale) Mirror 8: 0x6720ac73 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e55d: skipped 000233 1787894401.883734 MIRROR_RESYNC 5 mirrors 000235 1787894403.877410 write 0x02e101 thru 0x033dfc (0x5cfc bytes) 000237 1787894403.942767 trunc from 0x02545e to 0x032cb8 (0xd85b bytes) 000238 1787894403.961499 trunc from 0x03e55d to 0x038bc9 (0x5993 bytes) 000239 1787894403.989756 punch from 0x01fadc to 0x02c212 (0xc737 bytes) 000240 1787894404.002480 trunc from 0x015b3b to 0x01ece2 (0x91a8 bytes) 000241 1787894404.021203 write 0x005d7d thru 0x0110fd (0xb381 bytes) 000242 1787894404.046011 trunc from 0x038bca to 0x02175f (0x1746a bytes) 000243 1787894404.082687 mapread 0x00b5b2 thru 0x00c940 (0x138f bytes) 000244 1787894404.085639 punch from 0x013a7c to 0x019a11 (0x5f96 bytes) 000245 1787894404.100498 MIRROR_RESYNC 5 mirrors 000246 1787894405.107148 trunc from 0x021760 to 0x034ada (0x1337b bytes) 000247 1787894405.203253 mapread 0x01668e thru 0x022544 (0xbeb7 bytes) 000248 1787894405.242544 punch from 0x00aa06 to 0x00af6b (0x566 bytes) 000249 1787894405.248534 trunc from 0x034adb to 0x03d9c8 (0x8eee bytes) 000250 1787894405.273375 trunc from 0x00c317 to 0x014afd (0x87e7 bytes) 000251 1787894405.280866 fallocate 0x00294e thru 0x0122be (0xf971 bytes) 000252 1787894405.288770 trunc from 0x0214b9 to 0x02b066 (0x9bae bytes) 000253 1787894405.297702 write 0x03a560 thru 0x03ffff (0x5aa0 bytes) fallocating to largest ever: 0x40000 000254 1787894405.304906 fallocate 0x03cf60 thru 0x03ffff (0x30a0 bytes) 000255 1787894405.318430 mapwrite 0x036f8f thru 0x03cfd2 (0x6044 bytes) 000257 1787894405.392016 mapwrite 0x02c340 thru 0x033e1f (0x7ae0 bytes) 000258 1787894405.438153 trunc from 0x0103f8 to 0x014ff2 (0x4bfb bytes) 000259 1787894405.446920 write 0x027a4c thru 0x029b19 (0x20ce bytes) 000260 1787894405.448124 read 0x018c73 thru 0x01da27 (0x4db5 bytes) 000262 1787894405.450427 read 0x006e78 thru 0x00d58c (0x6715 bytes) 000264 1787894405.483515 read 0x00b8c7 thru 0x00e7c8 (0x2f02 bytes) 000265 1787894405.484198 mapread 0x02bcec thru 0x035e16 (0xa12b bytes) 000266 1787894405.490737 punch from 0x026eeb to 0x03343b (0xc551 bytes) 000267 1787894405.519424 fallocate 0x0347fb thru 0x03576c (0xf72 bytes) 000269 1787894405.537841 trunc from 0x006063 to 0x014a78 (0xea16 bytes) 000270 1787894405.548878 fallocate 0x01af54 thru 0x027736 (0xc7e3 bytes) 000271 1787894405.569380 read 0x0005b3 thru 0x006743 (0x6191 bytes) 000272 1787894405.621578 mapwrite 0x008f8c thru 0x00a3f7 (0x146c bytes) 000274 1787894405.642899 read 0x02f0be thru 0x036171 (0x70b4 bytes) 000275 1787894405.643772 trunc from 0x0165b9 to 0x020999 (0xa3e1 bytes) 000276 1787894405.654934 read 0x027fa9 thru 0x036bd5 (0xec2d bytes) 000277 1787894405.656544 write 0x026776 thru 0x0278c9 (0x1154 bytes) 000278 1787894405.657668 mapread 0x031ef3 thru 0x03dad0 (0xbbde bytes) 000279 1787894405.665787 write 0x0123ea thru 0x0206e6 (0xe2fd bytes) 000280 1787894405.671738 write 0x033b51 thru 0x03ffff (0xc4af bytes) 000282 1787894405.674164 trunc from 0x01a096 to 0x026b3d (0xcaa8 bytes) 000283 1787894405.693770 mapwrite 0x03bedf thru 0x03ffff (0x4121 bytes) 000284 1787894405.743367 mapwrite 0x02014c thru 0x028a07 (0x88bc bytes) 000285 1787894405.812128 mapread 0x0383fc thru 0x03ffff (0x7c04 bytes) 000286 1787894405.814438 write 0x03f544 thru 0x03ffff (0xabc bytes) 000287 1787894405.815997 trunc from 0x040000 to 0x03cfcc (0x3033 bytes) 000288 1787894405.840205 fallocate 0x02d51a thru 0x03a3a0 (0xce87 bytes) 000289 1787894405.852410 mapwrite 0x023782 thru 0x02d798 (0xa017 bytes) 000290 1787894405.889166 trunc from 0x03cfcd to 0x00844d (0x34b7f bytes) 000291 1787894405.916658 fallocate 0x03ffbe thru 0x03ffff (0x42 bytes) 000292 1787894405.923009 fallocate 0x01a2fb thru 0x02878b (0xe491 bytes) 000293 1787894405.929184 write 0x006281 thru 0x013856 (0xd5d6 bytes) 000294 1787894405.934919 write 0x0398cc thru 0x03ee87 (0x55bc bytes) 000295 1787894405.937682 punch from 0x013f32 to 0x018376 (0x4445 bytes) 000296 1787894405.961155 trunc from 0x02d42f to 0x02e7d2 (0x13a4 bytes) 000297 1787894405.967721 mapread 0x036fe6 thru 0x038ee8 (0x1f03 bytes) 000298 1787894405.992587 punch from 0x01b4b4 to 0x024cb3 (0x9800 bytes) 000299 1787894405.999590 mapwrite 0x033389 thru 0x03f856 (0xc4ce bytes) 000300 1787894406.073035 write 0x02226a thru 0x02f02f (0xcdc6 bytes) 000301 1787894406.085570 punch from 0x03cad1 to 0x03f856 (0x2d86 bytes) 000302 1787894406.094657 write 0x00d3ac thru 0x010f88 (0x3bdd bytes) 000303 1787894406.097522 trunc from 0x03b9ef to 0x03f856 (0x3e68 bytes) 000304 1787894406.107289 trunc from 0x01a2cb to 0x029f48 (0xfc7e bytes) 000305 1787894406.133469 MIRROR_RESYNC 5 mirrors 000306 1787894407.896091 fallocate 0x03c2ba thru 0x03ffff (0x3d46 bytes) 000307 1787894407.954263 trunc from 0x03df78 to 0x03ffff (0x2088 bytes) 000308 1787894407.959961 trunc from 0x009a69 to 0x0192b0 (0xf848 bytes) 000309 1787894407.969666 read 0x03e426 thru 0x03fd49 (0x1924 bytes) 000310 1787894407.982891 trunc from 0x01f5bb to 0x024f49 (0x598f bytes) 000311 1787894407.994246 write 0x00d43d thru 0x01d24e (0xfe12 bytes) 000312 1787894408.015813 read 0x013097 thru 0x01d66d (0xa5d7 bytes) 000313 1787894408.016739 write 0x03a634 thru 0x03ffff (0x59cc bytes) 000314 1787894408.059839 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x48d6349f (stale) Mirror 6: 0x48d6349f (stale) Mirror 7: 0x48d6349f (stale) Mirror 8: 0x48d6349f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x65178231 Mirror 9: 0x65178231 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000315 1787894409.297191 trunc from 0x03c1c5 to 0x03db56 (0x1992 bytes) 000316 1787894409.374967 fallocate 0x038a2e thru 0x038fb7 (0x58a bytes) 000317 1787894409.381583 trunc from 0x040000 to 0x018072 (0x27f8d bytes) 000318 1787894409.443186 write 0x011e22 thru 0x01b28b (0x946a bytes) 000319 1787894409.450890 write 0x022654 thru 0x02b131 (0x8ade bytes) 000320 1787894409.453594 trunc from 0x02b132 to 0x03effa (0x13ec9 bytes) 000321 1787894409.473733 trunc from 0x03effb to 0x017ded (0x2720d bytes) 000322 1787894409.514263 punch from 0x00b55d to 0x013129 (0x7bcd bytes) 000323 1787894409.549850 mapread 0x002bc5 thru 0x00f2f7 (0xc733 bytes) 000324 1787894409.567004 write 0x0041e4 thru 0x00ded0 (0x9ced bytes) 000325 1787894409.569296 write 0x011ca9 thru 0x014821 (0x2b79 bytes) 000326 1787894409.579920 MIRROR_RESYNC 6 mirrors 000327 1787894410.769435 fallocate 0x024d73 thru 0x03442e (0xf6bc bytes) 000328 1787894410.803237 fallocate 0x01d26c thru 0x020778 (0x350d bytes) 000330 1787894410.809683 mapwrite 0x030136 thru 0x036543 (0x640e bytes) 000331 1787894410.875948 trunc from 0x036544 to 0x00291f (0x33c24 bytes) 000332 1787894410.910128 punch from 0x001b5c to 0x00291f (0xdc4 bytes) 000333 1787894410.918017 trunc from 0x002920 to 0x02b7c0 (0x28ea1 bytes) 000334 1787894410.939467 fallocate 0x0261e9 thru 0x02bc89 (0x5aa1 bytes) 000335 1787894410.945808 trunc from 0x026450 to 0x026816 (0x3c7 bytes) 000336 1787894410.950461 read 0x025812 thru 0x0266fc (0xeeb bytes) 000337 1787894410.957439 trunc from 0x018e22 to 0x0195fa (0x7d9 bytes) 000338 1787894410.962921 trunc from 0x01d812 to 0x028bd9 (0xb3c8 bytes) 000339 1787894410.968320 mapwrite 0x0249dd thru 0x02ad50 (0x6374 bytes) 000340 1787894411.002981 trunc from 0x005896 to 0x00d2d9 (0x7a44 bytes) 000341 1787894411.007834 trunc from 0x02bc8a to 0x0045da (0x276af bytes) 000342 1787894411.024135 mapread 0x000067 thru 0x0045da (0x4574 bytes) 000343 1787894411.034029 fallocate 0x011406 thru 0x012509 (0x1104 bytes) 000344 1787894411.038943 punch from 0x00d3ad to 0x00fe95 (0x2ae9 bytes) 000345 1787894411.043100 trunc from 0x00c464 to 0x012509 (0x60a6 bytes) 000346 1787894411.047742 trunc from 0x008248 to 0x012509 (0xa2c2 bytes) 000347 1787894411.053361 read 0x00a437 thru 0x012509 (0x80d3 bytes) 000348 1787894411.068784 trunc from 0x01250a to 0x00b838 (0x6cd1 bytes) 000349 1787894411.087287 mapwrite 0x004b3c thru 0x00d542 (0x8a07 bytes) 000350 1787894411.136555 write 0x032acc thru 0x03df1d (0xb452 bytes) 000351 1787894411.139540 mapread 0x0008cd thru 0x00780a (0x6f3e bytes) 000352 1787894411.142288 fallocate 0x03b0e8 thru 0x03ffff (0x4f18 bytes) 000354 1787894411.148410 fallocate 0x034d92 thru 0x039b27 (0x4d96 bytes) 000355 1787894411.155114 fallocate 0x026dfa thru 0x0270f4 (0x2fb bytes) 000356 1787894411.166078 read 0x01095c thru 0x01725d (0x6902 bytes) 000357 1787894411.186678 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5d483821 (stale) Mirror 6: 0x5d483821 (stale) Mirror 7: 0x5d483821 (stale) Mirror 8: 0x5d483821 (stale) Mirror 9: 0x5d483821 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x66bc8fe1 Mirror 10: 0x66bc8fe1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000359 1787894412.334304 read 0x02fb18 thru 0x03853d (0x8a26 bytes) 000360 1787894412.350943 fallocate 0x0199fd thru 0x0255c9 (0xbbcd bytes) 000361 1787894412.404446 MIRROR_RESYNC 7 mirrors 000362 1787894414.170108 MIRROR_EXTEND to 8 mirrors 000363 1787894415.001531 read 0x03dc13 thru 0x03ffff (0x23ed bytes) 000364 1787894415.008389 mapread 0x01c25e thru 0x024476 (0x8219 bytes) 000365 1787894415.035898 write 0x031a79 thru 0x0341fe (0x2786 bytes) 000366 1787894415.128057 write 0x026a2b thru 0x027f73 (0x1549 bytes) 000367 1787894415.152140 write 0x018402 thru 0x019c53 (0x1852 bytes) 000368 1787894415.164056 mapread 0x006270 thru 0x0091a2 (0x2f33 bytes) 000369 1787894415.173589 read 0x01a49a thru 0x020a51 (0x65b8 bytes) 000370 1787894415.203659 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x66bc8fe1 (stale) Mirror 7: 0x66bc8fe1 (stale) Mirror 8: 0x66bc8fe1 (stale) Mirror 9: 0x66bc8fe1 (stale) Mirror 10: 0x66bc8fe1 (stale) Mirror 11: 0x66bc8fe1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x66bc8fe1 (stale) Mirror 7: 0x66bc8fe1 (stale) Mirror 8: 0x66bc8fe1 (stale) Mirror 9: 0x66bc8fe1 (stale) Mirror 10: 0x66bc8fe1 (stale) Mirror 11: 0x66bc8fe1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000371 1787894415.751339 trunc from 0x022d07 to 0x02ea50 (0xbd4a bytes) 000372 1787894415.758331 mapread 0x029ec2 thru 0x036e1f (0xcf5e bytes) 000373 1787894415.786242 trunc from 0x00da1f to 0x01ba05 (0xdfe7 bytes) 000374 1787894415.791907 punch from 0x004686 to 0x0075eb (0x2f66 bytes) 000375 1787894415.801697 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x66bc8fe1 (stale) Mirror 8: 0x66bc8fe1 (stale) Mirror 9: 0x66bc8fe1 (stale) Mirror 10: 0x66bc8fe1 (stale) Mirror 11: 0x66bc8fe1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x66bc8fe1 (stale) Mirror 8: 0x66bc8fe1 (stale) Mirror 9: 0x66bc8fe1 (stale) Mirror 10: 0x66bc8fe1 (stale) Mirror 11: 0x66bc8fe1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000376 1787894416.431167 trunc from 0x040000 to 0x03c5f2 (0x3a0d bytes) 000377 1787894416.467428 fallocate 0x01c01b thru 0x02a205 (0xe1eb bytes) 000378 1787894416.483712 fallocate 0x01a673 thru 0x01ffa5 (0x5933 bytes) 000379 1787894416.495378 fallocate 0x001126 thru 0x0079e2 (0x68bd bytes) 000380 1787894416.504802 write 0x01c118 thru 0x01f25e (0x3147 bytes) 000381 1787894416.526945 mapread 0x0021c0 thru 0x004a9a (0x28db bytes) 000382 1787894416.559038 mapread 0x01b9b7 thru 0x022a2f (0x7079 bytes) 000383 1787894416.585591 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c5f3): Mirror 6: 0xada4b530 (stale) Mirror 8: 0xada4b530 (stale) Mirror 10: 0xada4b530 (stale) Mirror 11: 0xada4b530 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5f3): Mirror 6: 0xada4b530 (stale) Mirror 8: 0xada4b530 (stale) Mirror 10: 0xada4b530 (stale) Mirror 11: 0xada4b530 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5f3: skipped 000384 1787894417.114372 punch from 0x02cfb2 to 0x034c28 (0x7c77 bytes) 000386 1787894417.136214 fallocate 0x02d6c5 thru 0x02ef6e (0x18aa bytes) 000388 1787894417.149795 mapwrite 0x01e206 thru 0x01e496 (0x291 bytes) 000389 1787894417.193105 write 0x016937 thru 0x01eb21 (0x81eb bytes) 000390 1787894417.226317 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c5f3): Mirror 6: 0xada4b530 (stale) Mirror 10: 0xada4b530 (stale) Mirror 11: 0xada4b530 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5f3): Mirror 6: 0xada4b530 (stale) Mirror 10: 0xada4b530 (stale) Mirror 11: 0xada4b530 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5f3: skipped 000391 1787894417.569397 write 0x01f23f thru 0x01fb59 (0x91b bytes) 000393 1787894417.579843 punch from 0x025968 to 0x02b8d5 (0x5f6e bytes) 000394 1787894417.584577 write 0x00ca44 thru 0x015803 (0x8dc0 bytes) 000396 1787894417.594650 mapread 0x014245 thru 0x016d0f (0x2acb bytes) 000397 1787894417.605024 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c5f3): Mirror 6: 0xada4b530 (stale) Mirror 10: 0xada4b530 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5f3): Mirror 6: 0xada4b530 (stale) Mirror 10: 0xada4b530 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5f3: skipped 000398 1787894418.010857 mapwrite 0x01f8d7 thru 0x02b26d (0xb997 bytes) 000399 1787894418.093138 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c5f3): Mirror 10: 0xada4b530 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5f3): Mirror 10: 0xada4b530 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5f3: skipped 000400 1787894418.447151 read 0x02f774 thru 0x0358be (0x614b bytes) 000401 1787894418.454989 trunc from 0x03c5f3 to 0x03bc33 (0x9bf bytes) 000402 1787894418.489857 write 0x028a43 thru 0x0323c5 (0x9983 bytes) 000404 1787894418.501066 mapwrite 0x005c36 thru 0x0074a5 (0x1870 bytes) 000405 1787894418.588934 read 0x02e2e8 thru 0x02fcce (0x19e7 bytes) 000406 1787894418.589571 fallocate 0x00b99d thru 0x014f51 (0x95b5 bytes) 000407 1787894418.599928 write 0x03bd68 thru 0x03ffff (0x4298 bytes) 000408 1787894418.627042 write 0x013a31 thru 0x01d47b (0x9a4b bytes) 000409 1787894418.628302 write 0x02ef3d thru 0x033665 (0x4729 bytes) 000410 1787894418.630894 mapwrite 0x00e89e thru 0x00eb6e (0x2d1 bytes) 000412 1787894418.662052 punch from 0x039b2f to 0x03ffff (0x64d1 bytes) 000413 1787894418.667673 fallocate 0x0003a7 thru 0x007ac9 (0x7723 bytes) 000414 1787894418.673888 punch from 0x0253cf to 0x029106 (0x3d38 bytes) 000415 1787894418.679843 fallocate 0x030e95 thru 0x03ffff (0xf16b bytes) 000416 1787894418.685897 trunc from 0x040000 to 0x03fa40 (0x5bf bytes) 000417 1787894418.702361 mapread 0x02f80a thru 0x03b5cc (0xbdc3 bytes) 000418 1787894418.717869 mapwrite 0x02556b thru 0x026d69 (0x17ff bytes) 000419 1787894418.760975 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) 000420 1787894418.935501 mapread 0x0162ef thru 0x02018a (0x9e9c bytes) 000421 1787894418.958941 read 0x03dedc thru 0x03fa40 (0x1b65 bytes) 000422 1787894418.967126 punch from 0x03c45a to 0x03fa40 (0x35e7 bytes) 000423 1787894418.973345 mapread 0x0166f4 thru 0x0212dc (0xabe9 bytes) 000424 1787894418.987234 fallocate 0x017e7f thru 0x022c93 (0xae15 bytes) 000426 1787894418.992895 mapread 0x0376ea thru 0x03fa40 (0x8357 bytes) 000428 1787894419.010155 mapwrite 0x023ed2 thru 0x028b8f (0x4cbe bytes) 000429 1787894419.032214 trunc from 0x03fa41 to 0x0330c8 (0xc978 bytes) 000430 1787894419.053499 read 0x02210a thru 0x025b7c (0x3a73 bytes) 000431 1787894419.054332 fallocate 0x03a0df thru 0x03ffff (0x5f21 bytes) 000432 1787894419.066124 trunc from 0x027841 to 0x02c8cf (0x508f bytes) 000433 1787894419.073419 fallocate 0x01e1d9 thru 0x02c10e (0xdf36 bytes) 000434 1787894419.084695 mapread 0x02461f thru 0x02493e (0x320 bytes) 000435 1787894419.087528 trunc from 0x040000 to 0x029197 (0x16e68 bytes) 000436 1787894419.110168 read 0x00243a thru 0x0066e8 (0x42af bytes) 000437 1787894419.129119 punch from 0x00dc9e to 0x018baf (0xaf12 bytes) 000438 1787894419.136553 mapwrite 0x010908 thru 0x010e94 (0x58d bytes) 000439 1787894419.167398 mapwrite 0x02dcfd thru 0x03513a (0x743e bytes) 000440 1787894419.249676 mapwrite 0x02c677 thru 0x031b2e (0x54b8 bytes) 000441 1787894419.279304 mapread 0x02699a thru 0x028ed2 (0x2539 bytes) 000442 1787894419.280232 read 0x00cd17 thru 0x015fab (0x9295 bytes) 000443 1787894419.292200 read 0x0257da thru 0x0289b2 (0x31d9 bytes) 000444 1787894419.293084 mapread 0x008e36 thru 0x017b61 (0xed2c bytes) 000445 1787894419.297304 write 0x03ba69 thru 0x03ffff (0x4597 bytes) 000446 1787894419.300159 trunc from 0x040000 to 0x03b1f4 (0x4e0b bytes) 000447 1787894419.323383 read 0x029070 thru 0x031ad0 (0x8a61 bytes) 000448 1787894419.324136 mapread 0x034b03 thru 0x03b1f4 (0x66f2 bytes) 000449 1787894419.333331 write 0x01916b thru 0x0278fe (0xe794 bytes) 000450 1787894419.334987 mapread 0x00236d thru 0x00dbe9 (0xb87d bytes) 000452 1787894419.342253 write 0x005ce0 thru 0x00738b (0x16ac bytes) 000453 1787894419.348815 trunc from 0x03b1f5 to 0x01dc84 (0x1d570 bytes) 000454 1787894419.367388 trunc from 0x01dc85 to 0x034fde (0x1735a bytes) 000455 1787894419.388562 read 0x01b7e8 thru 0x021559 (0x5d72 bytes) 000458 1787894419.407916 trunc from 0x013f75 to 0x022533 (0xe5bf bytes) 000459 1787894419.427871 write 0x019163 thru 0x01ebae (0x5a4c bytes) 000460 1787894419.439058 trunc from 0x018341 to 0x01f1ad (0x6e6d bytes) 000461 1787894419.454850 punch from 0x0204f6 to 0x022046 (0x1b51 bytes) 000462 1787894419.462421 trunc from 0x0144aa to 0x0165e9 (0x2140 bytes) 000464 1787894419.468841 mapwrite 0x035282 thru 0x03e05b (0x8dda bytes) 000465 1787894419.519686 write 0x00f8f9 thru 0x01d49b (0xdba3 bytes) 000466 1787894419.536094 read 0x03700c thru 0x03e05b (0x7050 bytes) 000469 1787894419.539006 mapwrite 0x0159fb thru 0x023911 (0xdf17 bytes) 000470 1787894419.580515 mapread 0x012b0e thru 0x013cb6 (0x11a9 bytes) 000471 1787894419.581581 fallocate 0x01b0a7 thru 0x024d6c (0x9cc6 bytes) 000472 1787894419.588715 trunc from 0x03e05c to 0x00f7eb (0x2e870 bytes) 000473 1787894419.633253 MIRROR_EXTEND to 2 mirrors 000474 1787894420.534766 read 0x0098be thru 0x00f7eb (0x5f2e bytes) 000475 1787894420.560393 mapwrite 0x016411 thru 0x022300 (0xbef0 bytes) 000476 1787894420.706553 write 0x005998 thru 0x015466 (0xfacf bytes) 000477 1787894420.713955 trunc from 0x022301 to 0x02183f (0xac1 bytes) 000478 1787894420.736824 write 0x00185f thru 0x003746 (0x1ee8 bytes) 000479 1787894420.745897 fallocate 0x00ae19 thru 0x00b86c (0xa54 bytes) 000480 1787894420.753787 fallocate 0x00a6b6 thru 0x019055 (0xe9a0 bytes) 000481 1787894420.759550 mapread 0x01cfaa thru 0x02183f (0x4896 bytes) 000482 1787894420.766023 MIRROR_RESYNC 2 mirrors 000483 1787894421.372757 read 0x00c673 thru 0x00f16f (0x2afd bytes) 000484 1787894421.380400 trunc from 0x021840 to 0x0380cf (0x16890 bytes) 000485 1787894421.429101 read 0x007ac2 thru 0x00e82c (0x6d6b bytes) 000486 1787894421.442934 read 0x0164e3 thru 0x0205f4 (0xa112 bytes) 000488 1787894421.444128 read 0x01f3ae thru 0x029578 (0xa1cb bytes) 000489 1787894421.447698 trunc from 0x006022 to 0x00bd47 (0x5d26 bytes) 000490 1787894421.455011 fallocate 0x01a323 thru 0x01bba2 (0x1880 bytes) 000491 1787894421.470574 mapread 0x01666b thru 0x021814 (0xb1aa bytes) 000492 1787894421.471443 punch from 0x02b1ad to 0x02ebe3 (0x3a37 bytes) 000494 1787894421.482193 fallocate 0x03c110 thru 0x03ffff (0x3ef0 bytes) 000495 1787894421.499031 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) 000496 1787894421.647691 trunc from 0x03f397 to 0x03ffff (0xc69 bytes) 000497 1787894421.653440 read 0x007a34 thru 0x011a60 (0xa02d bytes) 000498 1787894421.665770 trunc from 0x02bde1 to 0x0323cf (0x65ef bytes) 000499 1787894421.670658 read 0x017607 thru 0x01d2f8 (0x5cf2 bytes) 000500 1787894421.681289 read 0x024f07 thru 0x032ada (0xdbd4 bytes) 000501 1787894421.682481 trunc from 0x040000 to 0x03ab98 (0x5467 bytes) 000502 1787894421.700275 trunc from 0x039bcd to 0x03ab98 (0xfcc bytes) 000504 1787894421.709335 MIRROR_EXTEND to 2 mirrors 000505 1787894422.540264 read 0x00756a thru 0x00c98f (0x5426 bytes) 000506 1787894422.566934 trunc from 0x03ab99 to 0x0023e5 (0x387b3 bytes) 000507 1787894422.608817 trunc from 0x000cdf to 0x0023e5 (0x1707 bytes) 000508 1787894422.615870 trunc from 0x0023e6 to 0x032860 (0x3047b bytes) 000509 1787894422.639570 read 0x01db08 thru 0x02d56a (0xfa63 bytes) 000510 1787894422.651395 mapread 0x026f6b thru 0x02f25c (0x82f2 bytes) 000512 1787894422.653350 fallocate 0x0375e9 thru 0x039e4e (0x2866 bytes) 000513 1787894422.662450 fallocate 0x009afb thru 0x014532 (0xaa38 bytes) 000515 1787894422.670831 mapread 0x026675 thru 0x02ef77 (0x8903 bytes) 000516 1787894422.673513 fallocate 0x027ccc thru 0x02a89f (0x2bd4 bytes) 000517 1787894422.679189 punch from 0x024134 to 0x025d41 (0x1c0e bytes) 000518 1787894422.687844 mapwrite 0x0398f6 thru 0x03ffff (0x670a bytes) 000519 1787894422.736651 fallocate 0x005448 thru 0x00a1b0 (0x4d69 bytes) 000520 1787894422.742564 mapwrite 0x001a11 thru 0x00a0c3 (0x86b3 bytes) 000521 1787894422.765530 mapread 0x0222f1 thru 0x02a671 (0x8381 bytes) 000522 1787894422.782216 punch from 0x0361bd to 0x0364bf (0x303 bytes) 000523 1787894422.786518 read 0x032a5b thru 0x03b824 (0x8dca bytes) 000524 1787894422.792317 write 0x016b94 thru 0x024658 (0xdac5 bytes) 000525 1787894422.795182 mapwrite 0x03d285 thru 0x03ffff (0x2d7b bytes) 000526 1787894422.813372 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) 000527 1787894422.938048 fallocate 0x0356b7 thru 0x039894 (0x41de bytes) 000528 1787894422.942838 read 0x03a1b1 thru 0x03f1ee (0x503e bytes) 000530 1787894422.951139 write 0x02af3d thru 0x035deb (0xaeaf bytes) 000531 1787894422.983306 punch from 0x0281e1 to 0x0304c0 (0x82e0 bytes) 000532 1787894423.044334 MIRROR_EXTEND to 2 mirrors 000533 1787894424.139738 mapread 0x01f871 thru 0x02aa2f (0xb1bf bytes) 000534 1787894424.193801 read 0x00ee04 thru 0x014609 (0x5806 bytes) 000535 1787894424.211303 mapwrite 0x00d17c thru 0x00e4d2 (0x1357 bytes) 000536 1787894424.306287 trunc from 0x015eb5 to 0x017848 (0x1994 bytes) 000537 1787894424.312528 trunc from 0x0364b5 to 0x03ffff (0x9b4b bytes) 000538 1787894424.317841 fallocate 0x006f2b thru 0x00de88 (0x6f5e bytes) 000539 1787894424.326298 read 0x007d7c thru 0x00edfd (0x7082 bytes) 000540 1787894424.327295 punch from 0x019482 to 0x027cef (0xe86e bytes) 000541 1787894424.332927 mapread 0x02caed thru 0x035d75 (0x9289 bytes) 000542 1787894424.344453 fallocate 0x01de1f thru 0x0227d1 (0x49b3 bytes) 000543 1787894424.348552 trunc from 0x028a5d to 0x037b1d (0xf0c1 bytes) 000544 1787894424.354070 fallocate 0x021903 thru 0x0283a9 (0x6aa7 bytes) 000545 1787894424.359291 read 0x036a55 thru 0x03ffff (0x95ab bytes) 000546 1787894424.373452 trunc from 0x040000 to 0x02afee (0x15011 bytes) 000547 1787894424.406225 write 0x008aa3 thru 0x00f042 (0x65a0 bytes) 000548 1787894424.415698 trunc from 0x02afef to 0x0252c7 (0x5d27 bytes) 000549 1787894424.443298 mapread 0x012335 thru 0x0219d1 (0xf69d bytes) 000551 1787894424.453969 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) 000552 1787894424.648482 punch from 0x02252f to 0x0252c7 (0x2d99 bytes) 000553 1787894424.656361 punch from 0x007e64 to 0x008d9f (0xf3c bytes) 000554 1787894424.663300 trunc from 0x0252c8 to 0x0120b9 (0x1320e bytes) 000555 1787894424.696547 MIRROR_EXTEND to 2 mirrors 000556 1787894425.356660 trunc from 0x00af08 to 0x00e6bc (0x37b5 bytes) 000557 1787894425.405516 punch from 0x002e65 to 0x002eb8 (0x54 bytes) 000558 1787894425.417387 mapwrite 0x02145b thru 0x02756b (0x6111 bytes) 000559 1787894425.480428 fallocate 0x034983 thru 0x035466 (0xae4 bytes) 000560 1787894425.487354 punch from 0x014e87 to 0x017850 (0x29ca bytes) 000562 1787894425.494946 write 0x00d8c1 thru 0x0142eb (0x6a2b bytes) 000563 1787894425.510005 MIRROR_RESYNC 2 mirrors 000566 1787894426.310538 MIRROR_RESYNC 2 mirrors 000567 1787894426.661492 read 0x00a2e9 thru 0x01373c (0x9454 bytes) 000568 1787894426.677013 read 0x0352be thru 0x035466 (0x1a9 bytes) 000569 1787894426.711736 MIRROR_EXTEND to 3 mirrors 000570 1787894427.722181 MIRROR_EXTEND to 4 mirrors 000571 1787894428.703802 write 0x02172f thru 0x024eda (0x37ac bytes) 000572 1787894428.752347 read 0x027fb9 thru 0x03339b (0xb3e3 bytes) 000573 1787894428.781585 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35467): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35467): Mirror 2: 0xc2495828 (stale) Mirror 3: 0xc2495828 (stale) Mirror 4: 0xc2495828 (stale) CRC-32 checksum value for chunk [0, 0x35467): Mirror 1: 0x56da853b Mirror 5: 0x56da853b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35467: skipped 000574 1787894429.864956 mapread 0x0042e3 thru 0x00c44d (0x816b bytes) 000575 1787894429.905740 write 0x010905 thru 0x02062f (0xfd2b bytes) 000576 1787894429.970973 mapread 0x01cf85 thru 0x028e59 (0xbed5 bytes) 000577 1787894429.989773 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35467): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35467): Mirror 2: 0xc2495828 (stale) Mirror 3: 0xc2495828 (stale) Mirror 4: 0xc2495828 (stale) Mirror 5: 0x56da853b (stale) CRC-32 checksum value for chunk [0, 0x35467): Mirror 1: 0xcda20a4c Mirror 6: 0xcda20a4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35467: skipped 000578 1787894431.211280 read 0x030cbb thru 0x034b77 (0x3ebd bytes) 000579 1787894431.252594 write 0x02dbfb thru 0x0329e4 (0x4dea bytes) 000580 1787894431.342545 read 0x01640b thru 0x01a7c4 (0x43ba bytes) 000581 1787894431.384274 read 0x01176f thru 0x01702a (0x58bc bytes) 000582 1787894431.384983 mapread 0x005418 thru 0x007765 (0x234e bytes) 000583 1787894431.386986 punch from 0x02bd12 to 0x02e8b9 (0x2ba8 bytes) 000584 1787894431.409648 trunc from 0x008d0f to 0x00f5ed (0x68df bytes) 000585 1787894431.427745 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35467): Mirror 3: 0xc2495828 (stale) Mirror 4: 0xc2495828 (stale) Mirror 5: 0x56da853b (stale) Mirror 6: 0xcda20a4c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35467): Mirror 3: 0xc2495828 (stale) Mirror 4: 0xc2495828 (stale) Mirror 5: 0x56da853b (stale) Mirror 6: 0xcda20a4c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35467: skipped 000586 1787894432.023454 MIRROR_RESYNC 5 mirrors 000588 1787894433.987191 read 0x007d7b thru 0x00f735 (0x79bb bytes) 000589 1787894434.011871 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) 000590 1787894434.451379 fallocate 0x011166 thru 0x01eea2 (0xdd3d bytes) 000592 1787894434.500388 write 0x032804 thru 0x03aaa4 (0x82a1 bytes) 000593 1787894434.518593 write 0x02b814 thru 0x036551 (0xad3e bytes) 000596 1787894434.540228 mapwrite 0x0070c7 thru 0x01293f (0xb879 bytes) 000597 1787894434.664577 mapread 0x01d1d8 thru 0x021038 (0x3e61 bytes) 000598 1787894434.678547 punch from 0x022353 to 0x02c73b (0xa3e9 bytes) 000599 1787894434.684884 mapread 0x02c0d5 thru 0x032154 (0x6080 bytes) 000600 1787894434.697576 MIRROR_RESYNC 4 mirrors 000601 1787894435.601487 read 0x028167 thru 0x03452c (0xc3c6 bytes) 000602 1787894435.617817 fallocate 0x0290c4 thru 0x02d2e7 (0x4224 bytes) 000603 1787894435.639598 punch from 0x006d05 to 0x00dea6 (0x71a2 bytes) 000604 1787894435.644869 write 0x010c29 thru 0x01c4b8 (0xb890 bytes) 000605 1787894435.654483 write 0x00b933 thru 0x01b39f (0xfa6d bytes) 000606 1787894435.662734 mapwrite 0x02fd3b thru 0x03b619 (0xb8df bytes) 000607 1787894435.737379 read 0x00216c thru 0x004a14 (0x28a9 bytes) 000610 1787894435.755538 read 0x036492 thru 0x036b85 (0x6f4 bytes) 000611 1787894435.756496 write 0x01d4d7 thru 0x028453 (0xaf7d bytes) 000612 1787894435.758275 trunc from 0x03b61a to 0x01bbca (0x1fa4f bytes) 000613 1787894435.775007 punch from 0x012083 to 0x016bce (0x4b4c bytes) 000614 1787894435.781136 read 0x01188a thru 0x016cb9 (0x5430 bytes) 000615 1787894435.788721 fallocate 0x023b7b thru 0x03097d (0xce03 bytes) 000616 1787894435.802875 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3097e): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3097e): Mirror 3: 0x25c24513 (stale) Mirror 5: 0x25c24513 (stale) Mirror 6: 0x25c24513 (stale) CRC-32 checksum value for chunk [0, 0x3097e): Mirror 1: 0xceeb1100 Mirror 7: 0xceeb1100 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3097e: skipped 000617 1787894436.801977 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3097e): Mirror 3: 0x25c24513 (stale) Mirror 5: 0x25c24513 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3097e): Mirror 3: 0x25c24513 (stale) Mirror 5: 0x25c24513 (stale) CRC-32 checksum value for chunk [0, 0x3097e): Mirror 1: 0xceeb1100 Mirror 7: 0xceeb1100 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3097e: skipped 000618 1787894437.693309 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3097e): Mirror 3: 0x25c24513 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3097e): Mirror 3: 0x25c24513 (stale) CRC-32 checksum value for chunk [0, 0x3097e): Mirror 1: 0xceeb1100 Mirror 7: 0xceeb1100 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3097e: skipped 000619 1787894438.241322 mapread 0x028612 thru 0x03097d (0x836c bytes) 000620 1787894438.258405 fallocate 0x00f01a thru 0x011069 (0x2050 bytes) 000621 1787894438.297842 mapwrite 0x02b825 thru 0x037358 (0xbb34 bytes) 000622 1787894438.369463 write 0x01809a thru 0x01b786 (0x36ed bytes) 000623 1787894438.378308 write 0x02c298 thru 0x02c5cb (0x334 bytes) 000624 1787894438.379937 punch from 0x025a7c to 0x02ac28 (0x51ad bytes) 000625 1787894438.385266 mapwrite 0x00e147 thru 0x01374b (0x5605 bytes) 000626 1787894438.409598 mapwrite 0x0271f1 thru 0x0316d4 (0xa4e4 bytes) 000627 1787894438.437989 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37359): Mirror 7: 0x698265b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37359): Mirror 7: 0x698265b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37359: skipped 000628 1787894438.775410 punch from 0x01e518 to 0x021be3 (0x36cc bytes) 000629 1787894438.779504 mapwrite 0x01f884 thru 0x0211fd (0x197a bytes) 000630 1787894438.807767 mapwrite 0x01b0e1 thru 0x021399 (0x62b9 bytes) 000631 1787894438.827109 MIRROR_RESYNC 2 mirrors 000632 1787894439.538535 punch from 0x006276 to 0x0080e5 (0x1e70 bytes) 000633 1787894439.583227 trunc from 0x037359 to 0x018c44 (0x1e714 bytes) 000634 1787894439.627575 mapread 0x014c74 thru 0x017b36 (0x2ec3 bytes) 000635 1787894439.664226 mapwrite 0x015e26 thru 0x02434b (0xe526 bytes) 000636 1787894439.765936 MIRROR_RESYNC 2 mirrors 000637 1787894440.433965 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) 000638 1787894440.706115 read 0x007a86 thru 0x00f6aa (0x7c25 bytes) 000639 1787894440.719263 fallocate 0x0351f3 thru 0x035e50 (0xc5e bytes) 000640 1787894440.750924 read 0x01bdec thru 0x028412 (0xc627 bytes) 000641 1787894440.769480 read 0x01e99a thru 0x0202f6 (0x195d bytes) 000642 1787894440.770087 punch from 0x01934a to 0x023ac5 (0xa77c bytes) 000644 1787894440.783068 trunc from 0x02d300 to 0x032b63 (0x5864 bytes) 000645 1787894440.792487 trunc from 0x035e51 to 0x005018 (0x30e38 bytes) 000646 1787894440.834365 trunc from 0x0019cb to 0x005018 (0x364e bytes) 000647 1787894440.841660 mapwrite 0x02b9d7 thru 0x039029 (0xd653 bytes) 000648 1787894440.902536 punch from 0x010d19 to 0x01b689 (0xa971 bytes) 000649 1787894440.906829 fallocate 0x010bef thru 0x0148c0 (0x3cd2 bytes) 000650 1787894440.912940 mapread 0x02c5fb thru 0x039029 (0xca2f bytes) 000653 1787894440.920951 MIRROR_EXTEND to 2 mirrors 000655 1787894441.962674 write 0x019564 thru 0x01e6df (0x517c bytes) 000656 1787894442.030081 read 0x0336c3 thru 0x035939 (0x2277 bytes) 000657 1787894442.037604 fallocate 0x03529f thru 0x038bdf (0x3941 bytes) 000658 1787894442.045551 trunc from 0x03902a to 0x01a6f0 (0x1e939 bytes) 000659 1787894442.098661 mapwrite 0x009000 thru 0x00ca2e (0x3a2f bytes) 000660 1787894442.185365 MIRROR_RESYNC 2 mirrors 000661 1787894443.132388 write 0x023d98 thru 0x0258c8 (0x1b31 bytes) 000662 1787894443.205351 fallocate 0x03b844 thru 0x03ffff (0x47bc bytes) 000663 1787894443.211719 fallocate 0x0068bb thru 0x0157b3 (0xeef9 bytes) 000664 1787894443.220593 trunc from 0x024665 to 0x0258c8 (0x1264 bytes) 000665 1787894443.244254 mapread 0x01b52f thru 0x01b711 (0x1e3 bytes) 000666 1787894443.252974 mapwrite 0x013b2d thru 0x01db50 (0xa024 bytes) 000667 1787894443.339440 write 0x00bb6e thru 0x0181e9 (0xc67c bytes) 000669 1787894443.349539 mapwrite 0x033bf3 thru 0x03ec93 (0xb0a1 bytes) 000670 1787894443.500883 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ec94): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ec94): Mirror 2: 0x53d5f7cb (stale) CRC-32 checksum value for chunk [0, 0x3ec94): Mirror 1: 0xd7d80c1d Mirror 3: 0xd7d80c1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ec94: skipped 000671 1787894444.789510 write 0x015d9a thru 0x022ea1 (0xd108 bytes) 000672 1787894444.840305 punch from 0x036ff9 to 0x038b90 (0x1b98 bytes) 000673 1787894444.845153 read 0x001cd1 thru 0x006167 (0x4497 bytes) 000674 1787894444.853993 mapread 0x03c44a thru 0x03ec93 (0x284a bytes) 000675 1787894444.892073 mapread 0x02655d thru 0x02d7fa (0x729e bytes) 000676 1787894444.931443 punch from 0x021662 to 0x02d511 (0xbeb0 bytes) 000678 1787894444.970970 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ec94): Mirror 3: 0xd7d80c1d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ec94): Mirror 3: 0xd7d80c1d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ec94: skipped 000679 1787894445.346166 trunc from 0x03ec94 to 0x01b352 (0x23941 bytes) 000680 1787894445.386749 write 0x014939 thru 0x01a4d7 (0x5b9f bytes) 000681 1787894445.404451 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) 000682 1787894445.734418 fallocate 0x0287c3 thru 0x02e68f (0x5ecd bytes) 000683 1787894445.744674 trunc from 0x014883 to 0x01c595 (0x7d13 bytes) 000684 1787894445.760570 mapread 0x01aa6f thru 0x02a010 (0xf5a2 bytes) 000685 1787894445.795164 mapread 0x016977 thru 0x023fd0 (0xd65a bytes) 000686 1787894445.796545 read 0x01dab6 thru 0x021e03 (0x434e bytes) 000687 1787894445.800749 mapwrite 0x01f5b2 thru 0x02ab6e (0xb5bd bytes) 000688 1787894445.861393 trunc from 0x02e690 to 0x0200f6 (0xe599 bytes) 000689 1787894445.891683 punch from 0x000c17 to 0x005348 (0x4732 bytes) 000690 1787894445.897259 fallocate 0x001598 thru 0x00ae57 (0x98c0 bytes) 000691 1787894445.905890 trunc from 0x0200f7 to 0x015d31 (0xa3c5 bytes) 000692 1787894445.928243 trunc from 0x015d32 to 0x015438 (0x8f9 bytes) 000693 1787894445.953306 trunc from 0x015439 to 0x030600 (0x1b1c8 bytes) 000694 1787894445.979083 mapwrite 0x01b92e thru 0x021064 (0x5737 bytes) 000696 1787894446.048479 mapwrite 0x032889 thru 0x03dc2d (0xb3a5 bytes) 000697 1787894446.139362 mapwrite 0x01d3f3 thru 0x025575 (0x8183 bytes) 000698 1787894446.198706 mapread 0x00c669 thru 0x01c3fa (0xfd92 bytes) 000700 1787894446.242229 read 0x01a37e thru 0x024622 (0xa2a5 bytes) 000701 1787894446.243396 mapread 0x0121ea thru 0x016774 (0x458b bytes) 000702 1787894446.252145 mapwrite 0x026c3e thru 0x028f60 (0x2323 bytes) 000703 1787894446.279532 punch from 0x0122f3 to 0x01e867 (0xc575 bytes) 000704 1787894446.311456 fallocate 0x027213 thru 0x032243 (0xb031 bytes) 000705 1787894446.326161 write 0x0151e8 thru 0x0158b3 (0x6cc bytes) 000706 1787894446.353538 read 0x02f2ea thru 0x037ef5 (0x8c0c bytes) 000708 1787894446.355024 trunc from 0x03dc2e to 0x030662 (0xd5cb bytes) 000709 1787894446.410240 trunc from 0x02debe to 0x030662 (0x27a5 bytes) 000710 1787894446.446472 punch from 0x011b49 to 0x01dea7 (0xc35f bytes) 000711 1787894446.465510 trunc from 0x021a71 to 0x022495 (0xa25 bytes) 000712 1787894446.477570 mapread 0x02248f thru 0x030662 (0xe1d4 bytes) 000713 1787894446.505086 write 0x038287 thru 0x03976e (0x14e8 bytes) 000714 1787894446.506168 mapread 0x01af8b thru 0x01ed95 (0x3e0b bytes) 000715 1787894446.511200 mapwrite 0x010af4 thru 0x01f355 (0xe862 bytes) 000716 1787894446.572775 punch from 0x0314da to 0x03976e (0x8295 bytes) 000717 1787894446.582771 mapread 0x00ed02 thru 0x011095 (0x2394 bytes) 000718 1787894446.585383 trunc from 0x03976f to 0x0298a3 (0xfecb bytes) 000719 1787894446.609392 read 0x014d34 thru 0x01ed7a (0xa047 bytes) 000720 1787894446.616393 MIRROR_EXTEND to 2 mirrors 000721 1787894447.460975 read 0x015366 thru 0x0218f5 (0xc590 bytes) 000722 1787894447.483352 trunc from 0x0298a4 to 0x003778 (0x2612b bytes) 000723 1787894447.548842 trunc from 0x003779 to 0x01f70c (0x1bf94 bytes) 000724 1787894447.569121 punch from 0x013e8c to 0x016f52 (0x30c7 bytes) 000725 1787894447.574057 mapwrite 0x000486 thru 0x00ddcc (0xd947 bytes) 000727 1787894447.619545 trunc from 0x0146e0 to 0x0166c3 (0x1fe4 bytes) 000728 1787894447.628180 mapread 0x01c630 thru 0x01f70c (0x30dd bytes) 000729 1787894447.649404 fallocate 0x01b04c thru 0x020c71 (0x5c26 bytes) 000730 1787894447.665200 write 0x030fd8 thru 0x039306 (0x832f bytes) 000731 1787894447.667397 punch from 0x029448 to 0x0338f0 (0xa4a9 bytes) 000732 1787894447.687271 write 0x03b98c thru 0x03ffff (0x4674 bytes) 000733 1787894447.688369 trunc from 0x040000 to 0x03abbd (0x5442 bytes) 000734 1787894447.706335 punch from 0x01ffd3 to 0x02f53e (0xf56c bytes) 000735 1787894447.712921 trunc from 0x03abbe to 0x03ef76 (0x43b9 bytes) 000736 1787894447.731932 write 0x005ba3 thru 0x01273f (0xcb9d bytes) 000737 1787894447.738436 read 0x02e031 thru 0x037607 (0x95d7 bytes) 000739 1787894447.748213 trunc from 0x003135 to 0x00fc9e (0xcb6a bytes) 000740 1787894447.779456 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ef77): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef77): Mirror 2: 0x30850364 (stale) CRC-32 checksum value for chunk [0, 0x3ef77): Mirror 1: 0x22e8cb4 Mirror 3: 0x22e8cb4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef77: skipped 000741 1787894448.741198 write 0x028c0a thru 0x02fd7b (0x7172 bytes) 000742 1787894448.830532 read 0x0341dd thru 0x03a249 (0x606d bytes) 000743 1787894448.841829 mapread 0x00b907 thru 0x010945 (0x503f bytes) 000744 1787894448.869241 trunc from 0x03ef77 to 0x035149 (0x9e2d bytes) 000745 1787894448.895246 MIRROR_RESYNC 3 mirrors 000746 1787894449.706744 fallocate 0x027ad5 thru 0x036fff (0xf52b bytes) 000747 1787894449.747385 trunc from 0x0286f6 to 0x035676 (0xcf81 bytes) 000748 1787894449.752425 punch from 0x009374 to 0x00df8c (0x4c19 bytes) 000749 1787894449.757367 read 0x006d02 thru 0x014cdc (0xdfdb bytes) 000750 1787894449.774125 punch from 0x01bee7 to 0x02941b (0xd535 bytes) 000751 1787894449.779973 read 0x0211f2 thru 0x023476 (0x2285 bytes) 000752 1787894449.789321 trunc from 0x009079 to 0x013d37 (0xacbf bytes) 000753 1787894449.794579 mapwrite 0x02fea7 thru 0x036c3b (0x6d95 bytes) 000754 1787894449.820398 fallocate 0x00f5a6 thru 0x010789 (0x11e4 bytes) 000755 1787894449.826072 fallocate 0x021d2f thru 0x02c2c8 (0xa59a bytes) 000756 1787894449.833386 write 0x013200 thru 0x015b31 (0x2932 bytes) 000758 1787894449.839574 punch from 0x0091bc to 0x012539 (0x937e bytes) 000759 1787894449.853815 write 0x008e24 thru 0x01514a (0xc327 bytes) 000760 1787894449.858538 fallocate 0x02294e thru 0x024752 (0x1e05 bytes) 000761 1787894449.867303 trunc from 0x01e745 to 0x02ccd2 (0xe58e bytes) 000762 1787894449.877051 read 0x01ee60 thru 0x020319 (0x14ba bytes) 000763 1787894449.905698 trunc from 0x031658 to 0x031c8e (0x637 bytes) 000764 1787894449.917077 trunc from 0x037000 to 0x0104b0 (0x26b4f bytes) 000765 1787894449.967487 mapwrite 0x01df95 thru 0x02b649 (0xd6b5 bytes) 000766 1787894450.101405 read 0x0127d7 thru 0x01c775 (0x9f9f bytes) 000767 1787894450.112464 mapread 0x00c318 thru 0x0105f0 (0x42d9 bytes) 000768 1787894450.118345 trunc from 0x02b64a to 0x0361de (0xab95 bytes) 000769 1787894450.134566 trunc from 0x03202c to 0x0361de (0x41b3 bytes) 000770 1787894450.143251 punch from 0x02a5d0 to 0x032868 (0x8299 bytes) 000771 1787894450.153407 punch from 0x02db93 to 0x033672 (0x5ae0 bytes) 000772 1787894450.166206 trunc from 0x01f6c1 to 0x021978 (0x22b8 bytes) 000773 1787894450.173446 punch from 0x005232 to 0x01300a (0xddd9 bytes) 000775 1787894450.186733 mapread 0x030e26 thru 0x033b9f (0x2d7a bytes) 000776 1787894450.201214 write 0x03e86c thru 0x03ffff (0x1794 bytes) 000777 1787894450.211658 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9818124f (stale) Mirror 3: 0x9818124f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfcb93f42 Mirror 4: 0xfcb93f42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000778 1787894451.166323 fallocate 0x002fb9 thru 0x005d9d (0x2de5 bytes) 000779 1787894451.212538 punch from 0x037e79 to 0x03ffff (0x8187 bytes) 000780 1787894451.218535 write 0x0189d5 thru 0x021bc9 (0x91f5 bytes) 000781 1787894451.232555 read 0x015580 thru 0x01b9cc (0x644d bytes) 000782 1787894451.253917 write 0x002825 thru 0x0081e3 (0x59bf bytes) 000783 1787894451.265720 trunc from 0x00ddc2 to 0x01564d (0x788c bytes) 000784 1787894451.276790 mapread 0x03c8e5 thru 0x03ffff (0x371b bytes) 000785 1787894451.299208 trunc from 0x01dd12 to 0x026b28 (0x8e17 bytes) 000786 1787894451.316329 punch from 0x02b5fc to 0x039d9e (0xe7a3 bytes) 000787 1787894451.327810 trunc from 0x040000 to 0x033cbc (0xc343 bytes) 000788 1787894451.418056 fallocate 0x001ec2 thru 0x003733 (0x1872 bytes) 000789 1787894451.423904 mapread 0x021265 thru 0x022628 (0x13c4 bytes) 000790 1787894451.433711 read 0x02662b thru 0x02a18e (0x3b64 bytes) 000791 1787894451.444125 trunc from 0x00b7bb to 0x0154fa (0x9d40 bytes) 000792 1787894451.451343 trunc from 0x02289e to 0x02e22d (0xb990 bytes) 000793 1787894451.457368 trunc from 0x01c730 to 0x02177a (0x504b bytes) 000794 1787894451.462526 mapread 0x00c819 thru 0x01a9f8 (0xe1e0 bytes) 000795 1787894451.528927 read 0x025d66 thru 0x029610 (0x38ab bytes) 000796 1787894451.538822 write 0x0005a6 thru 0x00db75 (0xd5d0 bytes) 000797 1787894451.541026 mapread 0x008db5 thru 0x0149e5 (0xbc31 bytes) 000798 1787894451.542388 read 0x01e2ec thru 0x02d9d2 (0xf6e7 bytes) 000799 1787894451.543343 read 0x02504e thru 0x02b95d (0x6910 bytes) 000800 1787894451.544224 trunc from 0x033cbd to 0x0150e7 (0x1ebd5 bytes) 000801 1787894451.559398 mapwrite 0x0171e6 thru 0x01b6f7 (0x4512 bytes) 000802 1787894451.612458 punch from 0x009240 to 0x0133c3 (0xa184 bytes) 000803 1787894451.618443 trunc from 0x0058d6 to 0x00ad16 (0x5441 bytes) 000804 1787894451.623881 mapread 0x0176e0 thru 0x01a661 (0x2f82 bytes) 000805 1787894451.625874 fallocate 0x025e08 thru 0x02a44a (0x4643 bytes) 000806 1787894451.630876 fallocate 0x039b4c thru 0x03ffff (0x64b4 bytes) 000807 1787894451.636326 mapread 0x00ab64 thru 0x0143c5 (0x9862 bytes) 000808 1787894451.696873 mapwrite 0x02152a thru 0x02e465 (0xcf3c bytes) 000809 1787894451.714752 fallocate 0x008a90 thru 0x00ce09 (0x437a bytes) 000810 1787894451.720189 punch from 0x010950 to 0x012988 (0x2039 bytes) 000811 1787894451.724131 write 0x00debb thru 0x00e710 (0x856 bytes) 000812 1787894451.725136 write 0x0393fa thru 0x03ffff (0x6c06 bytes) 000813 1787894451.727017 read 0x01efa8 thru 0x02d2dd (0xe336 bytes) 000814 1787894451.727936 mapread 0x00f563 thru 0x0190c3 (0x9b61 bytes) 000815 1787894451.740087 write 0x01b6c0 thru 0x0211e9 (0x5b2a bytes) 000817 1787894451.743381 punch from 0x01f6fa to 0x023ee7 (0x47ee bytes) 000819 1787894451.775099 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9818124f (stale) Mirror 3: 0x9818124f (stale) Mirror 4: 0xfcb93f42 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa76f92fc Mirror 5: 0xa76f92fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000820 1787894453.150735 trunc from 0x040000 to 0x030ba6 (0xf459 bytes) 000821 1787894453.307360 write 0x006a90 thru 0x00e182 (0x76f3 bytes) 000822 1787894453.347454 punch from 0x01c935 to 0x020d43 (0x440f bytes) 000823 1787894453.354813 trunc from 0x030ba7 to 0x00f059 (0x21b4d bytes) 000824 1787894453.392576 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xf05a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf05a): Mirror 2: 0xab7fb9a7 (stale) Mirror 3: 0xab7fb9a7 (stale) Mirror 4: 0xab7fb9a7 (stale) Mirror 5: 0xcb38a2b3 (stale) CRC-32 checksum value for chunk [0, 0xf05a): Mirror 1: 0xeb40b405 Mirror 6: 0xeb40b405 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf05a: skipped 000825 1787894454.243100 trunc from 0x00f05a to 0x019a71 (0xaa18 bytes) 000827 1787894454.322534 punch from 0x018e4e to 0x019467 (0x61a bytes) 000828 1787894454.327430 punch from 0x00ab46 to 0x00b494 (0x94f bytes) 000829 1787894454.332818 trunc from 0x019a72 to 0x00a80d (0xf264 bytes) 000830 1787894454.367402 trunc from 0x007c4a to 0x00a80d (0x2bc4 bytes) 000831 1787894454.375201 punch from 0x000657 to 0x00a80d (0xa1b7 bytes) 000832 1787894454.383814 trunc from 0x00a80e to 0x012a32 (0x8225 bytes) 000834 1787894454.405578 fallocate 0x020cdf thru 0x026d00 (0x6022 bytes) 000835 1787894454.418883 fallocate 0x00ca67 thru 0x01a190 (0xd72a bytes) 000836 1787894454.427558 write 0x03e334 thru 0x03ffff (0x1ccc bytes) 000837 1787894454.429286 trunc from 0x040000 to 0x006523 (0x39adc bytes) 000838 1787894454.446587 write 0x006b3c thru 0x00b560 (0x4a25 bytes) 000839 1787894454.455306 trunc from 0x006c89 to 0x00b560 (0x48d8 bytes) 000840 1787894454.474344 read 0x008f39 thru 0x00b560 (0x2628 bytes) 000842 1787894454.487381 trunc from 0x00b561 to 0x00b2c6 (0x29a bytes) 000843 1787894454.518515 fallocate 0x008e8b thru 0x00b502 (0x2678 bytes) 000844 1787894454.532388 fallocate 0x0386f2 thru 0x03ffff (0x790e bytes) 000845 1787894454.543574 fallocate 0x00bb51 thru 0x019177 (0xd627 bytes) 000846 1787894454.554129 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb2c7): Mirror 2: 0xe7178895 (stale) Mirror 3: 0xe7178895 (stale) Mirror 5: 0xbe741c7a (stale) Mirror 6: 0x2ee8d794 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb2c7): Mirror 2: 0xe7178895 (stale) Mirror 3: 0xe7178895 (stale) Mirror 5: 0xbe741c7a (stale) Mirror 6: 0x2ee8d794 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb2c7: skipped 000847 1787894454.914717 punch from 0x0016d9 to 0x00b2c6 (0x9bee bytes) 000848 1787894454.920436 write 0x017af3 thru 0x01ce63 (0x5371 bytes) 000850 1787894454.934656 punch from 0x012a37 to 0x01ce63 (0xa42d bytes) 000851 1787894454.959989 fallocate 0x027919 thru 0x02afdc (0x36c4 bytes) 000853 1787894454.966819 trunc from 0x011722 to 0x01ce63 (0xb742 bytes) 000854 1787894454.976637 trunc from 0x0173d8 to 0x019dd7 (0x2a00 bytes) 000856 1787894454.983484 trunc from 0x002fdd to 0x00b85d (0x8881 bytes) 000857 1787894454.993041 trunc from 0x01ce64 to 0x00a27a (0x12be9 bytes) 000858 1787894455.019776 read 0x004fe2 thru 0x0064a0 (0x14bf bytes) 000859 1787894455.029570 punch from 0x0040d6 to 0x0053a2 (0x12cd bytes) 000860 1787894455.035898 trunc from 0x0048da to 0x00a27a (0x59a1 bytes) 000862 1787894455.054894 trunc from 0x00a22c to 0x00a27a (0x4f bytes) 000863 1787894455.066105 fallocate 0x0044e9 thru 0x00db79 (0x9691 bytes) 000864 1787894455.076119 punch from 0x0015d3 to 0x002624 (0x1052 bytes) 000865 1787894455.089591 write 0x022abc thru 0x029875 (0x6dba bytes) 000866 1787894455.094810 punch from 0x01317d to 0x01a667 (0x74eb bytes) 000867 1787894455.103863 fallocate 0x037cd3 thru 0x0388f1 (0xc1f bytes) 000868 1787894455.116170 MIRROR_RESYNC 5 mirrors 000872 1787894456.226985 trunc from 0x01cea8 to 0x01ea14 (0x1b6d bytes) 000873 1787894456.299263 trunc from 0x004808 to 0x006efa (0x26f3 bytes) 000874 1787894456.304290 punch from 0x01f45f to 0x025b8f (0x6731 bytes) 000875 1787894456.310007 trunc from 0x029876 to 0x02087f (0x8ff6 bytes) 000876 1787894456.392885 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x20880): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20880): Mirror 2: 0xe060b4e (stale) Mirror 3: 0xe060b4e (stale) Mirror 5: 0xe060b4e (stale) Mirror 6: 0xe060b4e (stale) CRC-32 checksum value for chunk [0, 0x20880): Mirror 1: 0xe060b4e Mirror 7: 0xe060b4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20880: skipped 000877 1787894457.387498 mapread 0x0059f9 thru 0x007ff4 (0x25fc bytes) 000878 1787894457.425073 write 0x03bb61 thru 0x03ffff (0x449f bytes) 000879 1787894457.478304 punch from 0x029438 to 0x0353c0 (0xbf89 bytes) 000880 1787894457.485108 mapwrite 0x01c4d0 thru 0x0226a1 (0x61d2 bytes) 000881 1787894457.528714 fallocate 0x010b9f thru 0x013732 (0x2b94 bytes) 000882 1787894457.545361 mapread 0x009341 thru 0x012269 (0x8f29 bytes) 000883 1787894457.565406 trunc from 0x040000 to 0x003b6d (0x3c492 bytes) 000884 1787894457.586714 write 0x00c2c5 thru 0x01a8d7 (0xe613 bytes) 000885 1787894457.593914 fallocate 0x02a3e2 thru 0x03551a (0xb139 bytes) 000886 1787894457.606100 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3551b): Mirror 2: 0xb97984d1 (stale) Mirror 3: 0xb97984d1 (stale) Mirror 5: 0xb97984d1 (stale) Mirror 7: 0xe9161566 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3551b): Mirror 2: 0xb97984d1 (stale) Mirror 3: 0xb97984d1 (stale) Mirror 5: 0xb97984d1 (stale) Mirror 7: 0xe9161566 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3551b: skipped 000887 1787894458.023129 trunc from 0x03551b to 0x02c4be (0x905c bytes) 000888 1787894458.047133 read 0x0264a9 thru 0x028616 (0x216e bytes) 000889 1787894458.053662 mapread 0x0086e4 thru 0x0115ec (0x8f09 bytes) 000891 1787894458.077478 write 0x00a08f thru 0x016218 (0xc18a bytes) 000892 1787894458.083147 mapread 0x02bf36 thru 0x02c4be (0x589 bytes) 000894 1787894458.094492 mapread 0x016cf2 thru 0x02402c (0xd33b bytes) 000895 1787894458.110180 read 0x02356b thru 0x02c4be (0x8f54 bytes) 000896 1787894458.111473 mapread 0x019ae1 thru 0x01f300 (0x5820 bytes) 000897 1787894458.112394 mapread 0x0050b2 thru 0x00b253 (0x61a2 bytes) 000898 1787894458.114401 write 0x023d42 thru 0x03064e (0xc90d bytes) 000899 1787894458.116721 fallocate 0x030415 thru 0x030a2c (0x618 bytes) 000900 1787894458.133039 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30a2d): Mirror 3: 0x8baaf90d (stale) Mirror 5: 0x8baaf90d (stale) Mirror 7: 0xb585e8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30a2d): Mirror 3: 0x8baaf90d (stale) Mirror 5: 0x8baaf90d (stale) Mirror 7: 0xb585e8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30a2d: skipped 000901 1787894458.460089 mapread 0x016090 thru 0x018364 (0x22d5 bytes) 000902 1787894458.484209 mapwrite 0x00a56f thru 0x00e344 (0x3dd6 bytes) 000903 1787894458.510853 MIRROR_RESYNC 4 mirrors 000904 1787894459.398103 trunc from 0x030a2d to 0x024af2 (0xbf3a bytes) 000905 1787894459.450773 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24af3): Mirror 5: 0x5671920b (stale) Mirror 7: 0x5671920b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24af3): Mirror 5: 0x5671920b (stale) Mirror 7: 0x5671920b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24af3: skipped 000906 1787894459.827338 trunc from 0x024af3 to 0x002515 (0x225dd bytes) 000907 1787894459.854445 trunc from 0x002516 to 0x005810 (0x32fb bytes) 000908 1787894459.880875 fallocate 0x00e95c thru 0x016eab (0x8550 bytes) 000909 1787894459.887888 trunc from 0x0052ae to 0x005810 (0x563 bytes) 000910 1787894459.893846 read 0x000891 thru 0x005810 (0x4f80 bytes) 000911 1787894459.906869 punch from 0x00086b to 0x005810 (0x4fa6 bytes) 000912 1787894459.914907 write 0x001d57 thru 0x003802 (0x1aac bytes) 000913 1787894459.927039 punch from 0x0031c9 to 0x005810 (0x2648 bytes) 000914 1787894459.940857 read 0x001a3a thru 0x005810 (0x3dd7 bytes) 000915 1787894459.950374 trunc from 0x005811 to 0x00a266 (0x4a56 bytes) 000916 1787894459.971444 mapread 0x003148 thru 0x005c57 (0x2b10 bytes) 000917 1787894459.973401 trunc from 0x00a267 to 0x00db3f (0x38d9 bytes) 000918 1787894459.993841 write 0x02b22f thru 0x02eff9 (0x3dcb bytes) 000919 1787894459.997544 punch from 0x01cad3 to 0x0288a0 (0xbdce bytes) 000920 1787894460.003474 trunc from 0x025c1f to 0x02eff9 (0x93db bytes) 000921 1787894460.021873 mapwrite 0x03517c thru 0x03cd5c (0x7be1 bytes) 000922 1787894460.154447 trunc from 0x0078db to 0x0144e4 (0xcc0a bytes) 000923 1787894460.169377 mapwrite 0x015f79 thru 0x0208f2 (0xa97a bytes) 000924 1787894460.257198 MIRROR_RESYNC 3 mirrors 000925 1787894461.460391 trunc from 0x01a69c to 0x01b335 (0xc9a bytes) 000926 1787894461.501292 read 0x01d33f thru 0x01e8f1 (0x15b3 bytes) 000927 1787894461.507952 trunc from 0x03cd5d to 0x007d9d (0x34fbf bytes) 000928 1787894461.526044 mapwrite 0x00fbbb thru 0x0107ab (0xbf1 bytes) 000929 1787894461.573332 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x107ac): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x107ac): Mirror 5: 0x649aaf40 (stale) Mirror 7: 0x649aaf40 (stale) CRC-32 checksum value for chunk [0, 0x107ac): Mirror 1: 0x99ec1aa6 Mirror 8: 0x99ec1aa6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x107ac: skipped 000930 1787894462.417075 write 0x02cc9e thru 0x03414d (0x74b0 bytes) 000931 1787894462.473148 trunc from 0x03414e to 0x01c83a (0x17913 bytes) 000932 1787894462.547705 fallocate 0x002009 thru 0x00fa96 (0xda8e bytes) 000933 1787894462.560309 fallocate 0x024f2f thru 0x032c52 (0xdd24 bytes) 000934 1787894462.569049 punch from 0x01a8da to 0x01c83a (0x1f61 bytes) 000935 1787894462.591101 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1c83b): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c83b): Mirror 5: 0x431fbe62 (stale) Mirror 7: 0x431fbe62 (stale) Mirror 8: 0x34a8592a (stale) CRC-32 checksum value for chunk [0, 0x1c83b): Mirror 1: 0x34a8592a Mirror 9: 0x34a8592a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c83b: skipped 000936 1787894463.900294 punch from 0x009d2a to 0x00cb6d (0x2e44 bytes) 000939 1787894463.964275 trunc from 0x01c83b to 0x027d1b (0xb4e1 bytes) 000940 1787894463.989155 trunc from 0x027d1c to 0x025de0 (0x1f3b bytes) 000941 1787894464.005293 punch from 0x01400e to 0x016ab6 (0x2aa9 bytes) 000943 1787894464.010792 trunc from 0x000af0 to 0x0101d4 (0xf6e5 bytes) 000944 1787894464.015951 fallocate 0x01f595 thru 0x02f4d3 (0xff3f bytes) 000947 1787894464.023003 write 0x03aac8 thru 0x03ffff (0x5538 bytes) 000948 1787894464.024544 trunc from 0x02c4c7 to 0x032f6e (0x6aa8 bytes) 000949 1787894464.029431 trunc from 0x00d8b3 to 0x0131a4 (0x58f2 bytes) 000950 1787894464.038331 mapwrite 0x03f4fe thru 0x03ffff (0xb02 bytes) 000951 1787894464.073643 trunc from 0x040000 to 0x03b18e (0x4e71 bytes) 000952 1787894464.097014 read 0x034cb4 thru 0x036a2f (0x1d7c bytes) 000953 1787894464.111738 write 0x0300fe thru 0x039582 (0x9485 bytes) 000954 1787894464.138645 punch from 0x0348c9 to 0x03b18e (0x68c6 bytes) 000955 1787894464.165841 trunc from 0x03b18f to 0x00e316 (0x2ce78 bytes) 000956 1787894464.190593 trunc from 0x0012cf to 0x0038a2 (0x25d4 bytes) 000957 1787894464.208293 mapwrite 0x03cf5b thru 0x03ffff (0x30a5 bytes) 000958 1787894464.288767 mapwrite 0x03a1cf thru 0x03d856 (0x3688 bytes) 000959 1787894464.306168 fallocate 0x00917b thru 0x01807c (0xef02 bytes) 000960 1787894464.329495 mapread 0x0208f9 thru 0x020934 (0x3c bytes) 000961 1787894464.345297 punch from 0x00b149 to 0x011c13 (0x6acb bytes) 000962 1787894464.350355 mapwrite 0x0326d5 thru 0x03b3e6 (0x8d12 bytes) 000963 1787894464.393010 MIRROR_RESYNC 5 mirrors 000964 1787894465.565025 MIRROR_RESYNC 5 mirrors 000965 1787894466.002378 trunc from 0x040000 to 0x00cd4c (0x332b3 bytes) 000966 1787894466.079107 trunc from 0x00cbd0 to 0x00cd4c (0x17d bytes) 000967 1787894466.084444 mapread 0x003409 thru 0x006c78 (0x3870 bytes) 000968 1787894466.101430 punch from 0x00812b to 0x00cd4c (0x4c22 bytes) 000969 1787894466.106334 mapread 0x0004c7 thru 0x009a93 (0x95cd bytes) 000970 1787894466.115645 punch from 0x00af8d to 0x00cd4c (0x1dc0 bytes) 000971 1787894466.120367 write 0x0290d1 thru 0x02ce69 (0x3d99 bytes) 000972 1787894466.122345 write 0x0073b5 thru 0x00876e (0x13ba bytes) 000973 1787894466.123918 trunc from 0x01d02b to 0x0210c6 (0x409c bytes) 000974 1787894466.132346 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ce6a): Mirror 7: 0x66cd7ad4 (stale) Mirror 8: 0x66cd7ad4 (stale) Mirror 9: 0x66cd7ad4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce6a): Mirror 7: 0x66cd7ad4 (stale) Mirror 8: 0x66cd7ad4 (stale) Mirror 9: 0x66cd7ad4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce6a: skipped 000975 1787894466.570357 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ce6a): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce6a): Mirror 7: 0x66cd7ad4 (stale) Mirror 8: 0x66cd7ad4 (stale) Mirror 9: 0x66cd7ad4 (stale) CRC-32 checksum value for chunk [0, 0x2ce6a): Mirror 1: 0xbe95cca1 Mirror 10: 0xbe95cca1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce6a: skipped 000976 1787894467.944787 read 0x028fb3 thru 0x02b9c6 (0x2a14 bytes) 000978 1787894467.967906 read 0x02b746 thru 0x02ce69 (0x1724 bytes) 000979 1787894468.004354 trunc from 0x02ce6a to 0x01a2e2 (0x12b87 bytes) 000980 1787894468.051521 trunc from 0x01a2e3 to 0x025197 (0xaeb5 bytes) 000981 1787894468.072105 punch from 0x01833c to 0x0209b9 (0x867e bytes) 000982 1787894468.076381 trunc from 0x00ca47 to 0x0166cb (0x9c85 bytes) 000983 1787894468.080671 trunc from 0x025198 to 0x014803 (0x10994 bytes) 000984 1787894468.096394 trunc from 0x008555 to 0x008df2 (0x89e bytes) 000985 1787894468.101403 punch from 0x0073a0 to 0x009ff8 (0x2c59 bytes) 000986 1787894468.107264 trunc from 0x012b8f to 0x014803 (0x1c75 bytes) 000988 1787894468.113186 mapwrite 0x01f44f thru 0x02af27 (0xbad9 bytes) 000989 1787894468.197752 write 0x023815 thru 0x029aec (0x62d8 bytes) 000990 1787894468.199362 trunc from 0x02af28 to 0x00adcc (0x2015b bytes) 000991 1787894468.216514 write 0x00517e thru 0x0143ab (0xf22e bytes) 000992 1787894468.224529 read 0x004c52 thru 0x00ad29 (0x60d8 bytes) 000993 1787894468.246667 trunc from 0x0143ac to 0x026e2d (0x12a82 bytes) 000994 1787894468.271322 punch from 0x00dbe2 to 0x012c42 (0x5061 bytes) 000995 1787894468.304346 trunc from 0x026e2e to 0x0213eb (0x5a42 bytes) 000996 1787894468.323600 trunc from 0x0213ec to 0x02af5f (0x9b74 bytes) 000997 1787894468.339077 mapread 0x01bd0a thru 0x020632 (0x4929 bytes) 000998 1787894468.349162 trunc from 0x02af60 to 0x039b16 (0xebb7 bytes) 000999 1787894468.363916 trunc from 0x036b10 to 0x039b16 (0x3007 bytes) 001000 1787894468.369277 write 0x017a99 thru 0x02498f (0xcef7 bytes) All operations completed A-OK! PASS 70a (103s) == sanity-flr test 71: check mirror extend parent fid ==== 01:21:14 (1787894474) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1fa:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 01:21:23 (1787894483) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141701 s, 14.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116609 s, 9.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 01:21:31 (1787894491) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre2 Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2140800 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7499808 .. failed Extending file size to 8025984 .. Extending file size to 8828416 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9252672 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8207584 .. Extending file size to 9089280 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61532 61533 61535 61537 61538 done 192.168.206.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre2 (opts:) 192.168.206.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 01:22:51 (1787894571) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre2 Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4730720 .. Extending file size to 7148928 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7495232 .. Extending file size to 7717728 .. Extending file size to 7200448 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8904512 .. Extending file size to 8188192 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8437920 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8794496 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63158 63160 63162 63163 63164 done 192.168.206.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre2 (opts:) 192.168.206.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 01:24:09 (1787894649) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2220 1269084 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18508 3588512 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24112 3582908 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42620 7171420 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2220 1269084 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18508 3588512 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24112 3582908 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42620 7171420 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116088 s, 18.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187624 s, 11.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9e6e9255 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 01:24:28 (1787894668) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131338 s, 16.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 01:24:35 (1787894675) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 01:24:42 (1787894682) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115952 s, 18.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x275:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x275:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x276:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 01:24:50 (1787894690) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.079778 s, 26.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 01:24:59 (1787894699) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137932 s, 15.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 01:25:32 (1787894732) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg623-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655877 s, 16.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } PASS 204d (50s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 01:26:22 (1787894782) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg623-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166216 s, 12.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 01:27:09 (1787894829) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg623-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg623-server: Pool lustre.test_204f created Adding targets to pool oleg623-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg623-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130325 s, 16.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg623-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg623-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg623-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 01:28:23 (1787894903) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 01:28:31 (1787894911) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 01:28:38 (1787894918) oleg623-server: Pool lustre.test_206 created Adding targets to pool oleg623-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg623-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg623-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg623-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg623-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 01:29:06 (1787894946) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812774 s, 12.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477115 s, 22.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x77a1ab7b Mirror 3: 0x77a1ab7b PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 01:29:17 (1787894957) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 01:29:21 (1787894961) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 01:29:24 (1787894964) Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.101155 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1200000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1990000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1990000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1990000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x13e0000 Read this (4 bytes): BBBB 192.168.206.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg623-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 01:30:00 (1787895000) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0312008 s, 263 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 01:30:07 (1787895007) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493179 s, 21.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x85 lmm_fid: [0x200000405:0x85:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611269 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-28 01:30:09.000000000 -0400 Modify: 2026-08-28 01:30:09.000000000 -0400 Change: 2026-08-28 01:30:11.000000000 -0400 Birth: 2026-08-28 01:30:09.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 01:30:16 (1787895016) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065651 s, 16.0 MB/s oleg623-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg623-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg623-server: O/280000401/d31/13f: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 01:30:41 (1787895041) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.162867 s, 251 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0573636 s, 71.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 28 01:30 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 28 01:30 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 01:30:50 (1787895050) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 01:30:59 (1787895059) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1822 sec =========== 01:31:09 (1787895069) === sanity-flr: start cleanup 01:31:11 (1787895071) === === sanity-flr: finish cleanup 01:31:22 (1787895082) ===