-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 03:17:11 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg606-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 03:17:29 (1787555849) === oleg606-client.virtnet: executing check_config_client /mnt/lustre oleg606-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg606-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff97f646d8d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff97f646d8d800.idle_timeout=debug disable quota as required oleg606-server: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 03:18:06 (1787555886) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8219] == sanity-flr test 0a: lfs mirror create with -N option == 03:18:08 (1787555888) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:18:24 (1787555904) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:18:27 (1787555907) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:18:31 (1787555911) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:18:56 (1787555936) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:18:59 (1787555939) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 03:19:03 (1787555943) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff97f646d8d800.stats=clear osc.lustre-OST0001-osc-ffff97f646d8d800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:19:11 (1787555951) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:19:22 (1787555962) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:19:32 (1787555972) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 03:19:35 (1787555975) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:19:43 (1787555983) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 03:19:52 (1787555992) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:20:00 (1787556000) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:20:08 (1787556008) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:20:15 (1787556015) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:20:25 (1787556025) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:20:31 (1787556031) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929053 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836221 s, 12.5 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:20:37 (1787556037) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.228996 s, 32.1 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 03:20:45 (1787556045) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:20:56 (1787556056) Stopping /mnt/lustre-ost1 (opts:) on oleg606-server oleg606-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97f646d8d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97f646d8d800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg606-server: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg606-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg606-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg606-server oleg606-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97f646d8d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97f646d8d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg606-server: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg606-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (135s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:23:11 (1787556191) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.307128 s, 20.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg606-server oleg606-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97f646d8d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97f646d8d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg606-server: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg606-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (37s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:23:48 (1787556228) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg606-server oleg606-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97f646d8d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97f646d8d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg606-server: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg606-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg606-server oleg606-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97f646d8d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97f646d8d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg606-server: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg606-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:25:15 (1787556315) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.10087 s, 16.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 03:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.957s user 0m0.001s sys 0m0.163s Stopping /mnt/lustre-ost1 (opts:) on oleg606-server oleg606-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff97f646d8d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff97f646d8d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.922s user 0m0.001s sys 0m0.168s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg606-server: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg606-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg606-server oleg606-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff97f646d8d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97f646d8d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.192s user 0m0.001s sys 0m0.183s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg606-server: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg606-server: oleg606-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg606-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg606-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff97f646d8d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff97f646d8d800.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:26:48 (1787556408) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:26:51 (1787556411) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:26:55 (1787556415) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 03:26:59 (1787556419) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 03:27:09 (1787556429) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.658375 s, 19.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:27:22 (1787556442) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396009 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477788 s, 21.9 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:27:29 (1787556449) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:27:37 (1787556457) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000957971 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000924226 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00086789 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00242309 s, 423 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 03:27:49 (1787556469) create test files with size 15743553 .. e949764845a15abda0cbb29e6bc88359 - 9b492867843c7b5de1711c2234299c23 - b834fd14da4d6cd2aac2a332d93dbc04 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg606-client.virtnet /mnt/lustre (opts:) Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 03:28:16 (1787556496) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:28:31 (1787556511) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:28:38 (1787556518) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774322 s, 13.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626717 s, 16.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 03:28:49 (1787556529) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0828174 s, 25.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.359188 s, 11.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205702 s, 10.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 03:29:07 (1787556547) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 03:29:11 (1787556551) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:29:15 (1787556555) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350718 s, 29.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 03:29:25 (1787556565) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.281095 s, 11.2 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:29:35 (1787556575) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:29:47 (1787556587) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.389043 s, 27.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg606-client.virtnet /mnt/lustre (opts:) Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:29:59 (1787556599) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff97f646eb5000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.315544 s, 33.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg606-client.virtnet /mnt/lustre (opts:) Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff97f652bbb800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:30:15 (1787556615) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:30:59 (1787556659) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (35s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:31:34 (1787556694) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 03:31:38 (1787556698) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.317358 s, 9.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.122115 s, 25.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd0fc3447 (nosync,stale) Mirror 2: 0xd0fc3447 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 70ce41fc93527bc1ffe9c12b433882ef - mirror 1 checksum: a6ee77a207d50d3fbc996da43a6a4930 - mirror 2 checksum: 70ce41fc93527bc1ffe9c12b433882ef - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T03:31:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T03:31:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 70ce41fc93527bc1ffe9c12b433882ef - mirror 1 checksum: a6ee77a207d50d3fbc996da43a6a4930 - mirror 2 checksum: a6ee77a207d50d3fbc996da43a6a4930 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T03:31:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T03:31:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc0:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 03:31:49 (1787556709) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463289 s, 22.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:31:57 (1787556717) Failing mds1 on oleg606-server Stopping /mnt/lustre-mds1 (opts:) on oleg606-server 03:32:10 (1787556730) shut down facet: mds1 facet_host: oleg606-server facet_failover_host: oleg606-server Failover mds1 to oleg606-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg606-server: oleg606-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg606-client: oleg606-server: ssh exited with exit code 1 Started lustre-MDT0000 03:32:44 (1787556764) targets are mounted 03:32:44 (1787556764) facet_failover done oleg606-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223337 s, 47.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc2:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:33:01 (1787556781) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0361551 s, 566 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0223326 s, 917 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:33:13 (1787556793) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0831936 s, 246 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.014404 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:33:25 (1787556805) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0151316 s, 1.1 MB/s PASS 50c (12s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:33:37 (1787556817) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0459251 s, 446 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:33:46 (1787556826) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0124569 s, 329 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0366419 s, 112 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:33:53 (1787556833) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:34:13 (1787556853) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:34:41 (1787556881) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:35:05 (1787556905) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:35:32 (1787556932) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 03:35:40 (1787556940) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57579 mirror split pid 57580 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57579 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57580 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 03:36:50 (1787557010) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7013 skipping zero size read skipping zero length zero range skipping zero length punch hole skipping zero length punch hole skipping zero length punch hole skipping zero size read skipping zero length zero range 000010 1787557013.526719 mapwrite 0x019489 thru 0x020b06 (0x767e bytes) 000011 1787557013.628117 write 0x00a744 thru 0x00be89 (0x1746 bytes) 000012 1787557013.641724 mapread 0x00d836 thru 0x017ac5 (0xa290 bytes) 000013 1787557013.660683 read 0x01e464 thru 0x020b06 (0x26a3 bytes) fallocating to largest ever: 0x37147 000015 1787557013.670694 fallocate 0x029510 thru 0x037146 (0xdc37 bytes) 000016 1787557013.680178 fallocate 0x01606d thru 0x01f345 (0x92d9 bytes) 000018 1787557013.695108 MIRROR_EXTEND to 2 mirrors 000019 1787557014.822854 mapread 0x01759c thru 0x01953e (0x1fa3 bytes) 000020 1787557014.872517 trunc from 0x037147 to 0x026e84 (0x102c2 bytes) 000021 1787557014.945527 read 0x00e701 thru 0x013508 (0x4e08 bytes) 000022 1787557014.973749 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) 000023 1787557015.182903 trunc from 0x026e85 to 0x018835 (0xe64f bytes) 000024 1787557015.207725 mapwrite 0x011fbb thru 0x016250 (0x4296 bytes) fallocating to largest ever: 0x40000 000025 1787557015.262875 fallocate 0x03d368 thru 0x03ffff (0x2c98 bytes) 000026 1787557015.276106 fallocate 0x0163f5 thru 0x01cb41 (0x674d bytes) 000027 1787557015.288964 trunc from 0x00302d to 0x003da6 (0xd7a bytes) 000028 1787557015.297332 punch from 0x01112e to 0x018835 (0x7708 bytes) 000029 1787557015.310532 MIRROR_EXTEND to 2 mirrors 000030 1787557016.074565 trunc from 0x018836 to 0x01179e (0x7097 bytes) 000031 1787557016.201532 trunc from 0x0087bd to 0x01179e (0x8fe2 bytes) 000032 1787557016.208285 mapwrite 0x00d439 thru 0x01d23d (0xfe05 bytes) 000033 1787557016.289093 mapread 0x00622f thru 0x00672d (0x4ff bytes) 000034 1787557016.299378 fallocate 0x0132ab thru 0x01df60 (0xacb6 bytes) 000035 1787557016.307534 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) 000036 1787557016.429137 write 0x026c92 thru 0x02fd1d (0x908c bytes) 000037 1787557016.435448 fallocate 0x0204f4 thru 0x021e1c (0x1929 bytes) 000038 1787557016.442361 punch from 0x011b51 to 0x01a160 (0x8610 bytes) 000039 1787557016.447627 mapwrite 0x007a00 thru 0x00e96f (0x6f70 bytes) 000040 1787557016.482513 write 0x00ebfb thru 0x010b92 (0x1f98 bytes) 000041 1787557016.483488 mapread 0x01c4b4 thru 0x02bbd0 (0xf71d bytes) 000043 1787557016.496712 mapwrite 0x026266 thru 0x02d387 (0x7122 bytes) 000044 1787557016.513503 fallocate 0x02867d thru 0x02c7d5 (0x4159 bytes) 000045 1787557016.518007 trunc from 0x02fd1e to 0x034bd9 (0x4ebc bytes) 000046 1787557016.534443 read 0x021dd9 thru 0x02facc (0xdcf4 bytes) 000047 1787557016.536303 write 0x034741 thru 0x03b453 (0x6d13 bytes) 000048 1787557016.543121 mapwrite 0x036c3a thru 0x037308 (0x6cf bytes) 000049 1787557016.557961 fallocate 0x01b4ac thru 0x020ff4 (0x5b49 bytes) 000050 1787557016.562524 mapread 0x005519 thru 0x0153e7 (0xfecf bytes) 000051 1787557016.568499 trunc from 0x00b10f to 0x019263 (0xe155 bytes) 000052 1787557016.574411 fallocate 0x00ce77 thru 0x015b2e (0x8cb8 bytes) 000053 1787557016.578849 punch from 0x0352f3 to 0x03b453 (0x6161 bytes) 000055 1787557016.584594 mapread 0x008f82 thru 0x0148ec (0xb96b bytes) 000056 1787557016.602684 mapwrite 0x032323 thru 0x03ffff (0xdcdd bytes) 000057 1787557016.674946 mapwrite 0x022068 thru 0x02e0f1 (0xc08a bytes) 000059 1787557016.704979 punch from 0x01648c to 0x020919 (0xa48e bytes) 000060 1787557016.710077 write 0x01a2ac thru 0x022c2e (0x8983 bytes) 000061 1787557016.716536 punch from 0x022553 to 0x02d6b2 (0xb160 bytes) 000062 1787557016.733259 punch from 0x00cf6a to 0x0195a9 (0xc640 bytes) 000063 1787557016.739015 mapread 0x021b74 thru 0x02f7c3 (0xdc50 bytes) 000064 1787557016.754592 trunc from 0x040000 to 0x0278cb (0x18734 bytes) 000065 1787557016.769548 read 0x018dc5 thru 0x022ea3 (0xa0df bytes) 000066 1787557016.789181 MIRROR_EXTEND to 2 mirrors 000067 1787557017.842098 fallocate 0x038ab3 thru 0x03ffff (0x754d bytes) 000068 1787557017.901983 trunc from 0x040000 to 0x020ea4 (0x1f15b bytes) 000069 1787557017.933222 write 0x017e14 thru 0x01e06f (0x625c bytes) 000070 1787557017.946832 fallocate 0x0226ed thru 0x02b348 (0x8c5c bytes) 000072 1787557017.981812 trunc from 0x02b349 to 0x01ab46 (0x10802 bytes) 000073 1787557018.005189 read 0x00d57d thru 0x018afc (0xb580 bytes) 000074 1787557018.016409 mapread 0x011dcb thru 0x0134b7 (0x16ed bytes) 000075 1787557018.017922 fallocate 0x0055f5 thru 0x00e0f4 (0x8b00 bytes) 000076 1787557018.030682 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ab47): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ab47): Mirror 2: 0xb66e58fa (stale) CRC-32 checksum value for chunk [0, 0x1ab47): Mirror 1: 0x95af74f0 Mirror 3: 0x95af74f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ab47: skipped 000077 1787557018.960365 read 0x0142b5 thru 0x015f49 (0x1c95 bytes) 000078 1787557018.979707 mapread 0x003f83 thru 0x010d98 (0xce16 bytes) 000079 1787557019.018057 fallocate 0x01173e thru 0x01e5e8 (0xceab bytes) 000080 1787557019.079556 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ab47): Mirror 3: 0x95af74f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ab47): Mirror 3: 0x95af74f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ab47: skipped 000081 1787557019.404413 mapwrite 0x00b7ab thru 0x017216 (0xba6c bytes) 000082 1787557019.476348 write 0x03bcf3 thru 0x03ffff (0x430d bytes) 000083 1787557019.482688 mapwrite 0x009bf1 thru 0x010732 (0x6b42 bytes) 000084 1787557019.503473 fallocate 0x015321 thru 0x01d12b (0x7e0b bytes) 000085 1787557019.508996 mapread 0x03a516 thru 0x03ffff (0x5aea bytes) 000086 1787557019.521039 mapread 0x030450 thru 0x037136 (0x6ce7 bytes) 000087 1787557019.531960 fallocate 0x020478 thru 0x025278 (0x4e01 bytes) 000088 1787557019.537535 fallocate 0x031b4e thru 0x03c5b8 (0xaa6b bytes) 000089 1787557019.544008 trunc from 0x040000 to 0x02d327 (0x12cd8 bytes) 000090 1787557019.562883 trunc from 0x02313f to 0x02d19d (0xa05f bytes) 000091 1787557019.570217 trunc from 0x010513 to 0x01ebf7 (0xe6e5 bytes) 000092 1787557019.576475 punch from 0x028cfe to 0x02d327 (0x462a bytes) 000093 1787557019.580357 trunc from 0x0176c7 to 0x025a92 (0xe3cc bytes) 000094 1787557019.586008 write 0x00fb30 thru 0x0169a2 (0x6e73 bytes) 000095 1787557019.592096 write 0x018466 thru 0x022cc8 (0xa863 bytes) 000096 1787557019.600708 trunc from 0x007534 to 0x01200d (0xaada bytes) 000097 1787557019.622547 trunc from 0x01528e to 0x019ada (0x484d bytes) 000098 1787557019.629076 trunc from 0x017bfa to 0x01ef74 (0x737b bytes) 000099 1787557019.634440 mapwrite 0x00d255 thru 0x0169bb (0x9767 bytes) 000100 1787557019.731416 trunc from 0x02d328 to 0x02eeb9 (0x1b92 bytes) 000101 1787557019.753505 MIRROR_RESYNC 2 mirrors 000102 1787557020.629419 mapread 0x004fea thru 0x010713 (0xb72a bytes) 000103 1787557020.647102 punch from 0x01dc0d to 0x02377b (0x5b6f bytes) 000104 1787557020.702584 mapread 0x021ceb thru 0x022488 (0x79e bytes) 000105 1787557020.712036 read 0x0259b4 thru 0x02d3bb (0x7a08 bytes) 000106 1787557020.719576 fallocate 0x03d94d thru 0x03ffff (0x26b3 bytes) 000107 1787557020.729052 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) 000108 1787557020.963630 read 0x035e9f thru 0x03c1e7 (0x6349 bytes) 000109 1787557020.982749 trunc from 0x02dacf to 0x032c2d (0x515f bytes) 000110 1787557020.991474 read 0x03e435 thru 0x03ffff (0x1bcb bytes) 000111 1787557021.008807 write 0x007019 thru 0x00f1bf (0x81a7 bytes) 000112 1787557021.018394 trunc from 0x040000 to 0x005181 (0x3ae7e bytes) 000113 1787557021.060500 mapread 0x003166 thru 0x005181 (0x201c bytes) 000114 1787557021.061388 fallocate 0x021a9d thru 0x022d91 (0x12f5 bytes) 000115 1787557021.067325 punch from 0x0046be to 0x006bf7 (0x253a bytes) 000116 1787557021.071954 mapread 0x007b92 thru 0x01361a (0xba89 bytes) 000117 1787557021.157241 trunc from 0x00458f to 0x007822 (0x3294 bytes) 000120 1787557021.164427 trunc from 0x000c79 to 0x003b0b (0x2e93 bytes) 000121 1787557021.170124 read 0x017303 thru 0x022102 (0xae00 bytes) 000122 1787557021.180504 trunc from 0x0135fb to 0x022d91 (0xf797 bytes) 000123 1787557021.186206 fallocate 0x0114f7 thru 0x01f0c4 (0xdbce bytes) 000124 1787557021.192003 write 0x031205 thru 0x03ffff (0xedfb bytes) 000125 1787557021.218755 MIRROR_EXTEND to 2 mirrors 000127 1787557021.763207 fallocate 0x033167 thru 0x03ffff (0xce99 bytes) 000128 1787557021.805517 fallocate 0x027937 thru 0x02aa05 (0x30cf bytes) 000129 1787557021.815671 write 0x035a52 thru 0x03ffff (0xa5ae bytes) 000130 1787557021.834078 mapread 0x0056c9 thru 0x00b343 (0x5c7b bytes) 000131 1787557021.891087 punch from 0x03e44e to 0x03ffff (0x1bb2 bytes) 000132 1787557021.919347 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) 000133 1787557022.116406 mapread 0x0033c4 thru 0x006f41 (0x3b7e bytes) 000134 1787557022.139382 trunc from 0x040000 to 0x02e7b6 (0x11849 bytes) 000136 1787557022.188027 trunc from 0x01ab69 to 0x01fb31 (0x4fc9 bytes) 000138 1787557022.194235 punch from 0x024757 to 0x02ba2d (0x72d7 bytes) 000140 1787557022.205434 write 0x02495e thru 0x034795 (0xfe38 bytes) 000141 1787557022.211725 punch from 0x0231da to 0x02f3cf (0xc1f6 bytes) 000142 1787557022.228899 punch from 0x00a7b0 to 0x00efbc (0x480d bytes) 000143 1787557022.237331 punch from 0x015d2a to 0x01f753 (0x9a2a bytes) 000144 1787557022.242641 trunc from 0x003023 to 0x0126e0 (0xf6be bytes) 000145 1787557022.247917 fallocate 0x03e49f thru 0x03ffff (0x1b61 bytes) 000146 1787557022.253007 mapread 0x01e13c thru 0x028ad5 (0xa99a bytes) 000147 1787557022.273071 mapwrite 0x02e6ac thru 0x031800 (0x3155 bytes) 000148 1787557022.302872 punch from 0x010fcb to 0x019d5f (0x8d95 bytes) 000149 1787557022.315796 fallocate 0x03a2ba thru 0x03d41a (0x3161 bytes) 000150 1787557022.323986 punch from 0x03cbe7 to 0x03d41a (0x834 bytes) 000151 1787557022.333238 read 0x026e9a thru 0x03661a (0xf781 bytes) 000152 1787557022.345872 trunc from 0x02bac5 to 0x02d36d (0x18a9 bytes) 000153 1787557022.359508 mapread 0x009933 thru 0x00dd73 (0x4441 bytes) 000156 1787557022.375641 mapwrite 0x00363d thru 0x0072b1 (0x3c75 bytes) 000157 1787557022.397061 trunc from 0x028da4 to 0x030fbe (0x821b bytes) 000158 1787557022.408082 write 0x00d00c thru 0x01584a (0x883f bytes) 000159 1787557022.418174 mapwrite 0x005726 thru 0x00b167 (0x5a42 bytes) 000160 1787557022.464831 mapread 0x002875 thru 0x006580 (0x3d0c bytes) 000161 1787557022.469441 punch from 0x0224ec to 0x025357 (0x2e6c bytes) 000162 1787557022.484182 mapwrite 0x02c165 thru 0x036027 (0x9ec3 bytes) 000163 1787557022.528013 punch from 0x0157ee to 0x023c30 (0xe443 bytes) 000164 1787557022.534516 write 0x030a98 thru 0x03495c (0x3ec5 bytes) 000166 1787557022.536592 read 0x01b0e0 thru 0x01cc21 (0x1b42 bytes) 000167 1787557022.551583 write 0x03d2e5 thru 0x03ffff (0x2d1b bytes) 000168 1787557022.555009 trunc from 0x0045f0 to 0x00b458 (0x6e69 bytes) 000169 1787557022.561657 mapwrite 0x03388a thru 0x039391 (0x5b08 bytes) 000170 1787557022.591118 write 0x02fb6f thru 0x03f014 (0xf4a6 bytes) 000172 1787557022.601966 punch from 0x0227e8 to 0x02dc69 (0xb482 bytes) 000173 1787557022.614089 punch from 0x02b0dd to 0x02c06f (0xf93 bytes) 000174 1787557022.621113 fallocate 0x013ba4 thru 0x021640 (0xda9d bytes) 000176 1787557022.645170 read 0x0168b1 thru 0x0245ac (0xdcfc bytes) 000177 1787557022.663100 punch from 0x00cd23 to 0x00df00 (0x11de bytes) 000178 1787557022.673871 punch from 0x00a55a to 0x00fd6a (0x5811 bytes) 000179 1787557022.679966 write 0x03b143 thru 0x03eae3 (0x39a1 bytes) 000181 1787557022.687691 mapread 0x001b5c thru 0x010808 (0xecad bytes) 000182 1787557022.725903 write 0x01ac53 thru 0x01bedf (0x128d bytes) 000183 1787557022.727353 read 0x01c8d4 thru 0x0249aa (0x80d7 bytes) 000184 1787557022.728041 mapwrite 0x002b6d thru 0x010ffa (0xe48e bytes) 000185 1787557022.747189 punch from 0x0095b5 to 0x0107a8 (0x71f4 bytes) 000188 1787557022.753404 fallocate 0x025254 thru 0x02968c (0x4439 bytes) 000189 1787557022.758375 fallocate 0x004089 thru 0x010abd (0xca35 bytes) 000190 1787557022.763275 mapwrite 0x01b2aa thru 0x02298f (0x76e6 bytes) 000191 1787557022.780756 mapwrite 0x02bf91 thru 0x02cea4 (0xf14 bytes) 000192 1787557022.793976 mapread 0x00c320 thru 0x017cdf (0xb9c0 bytes) 000193 1787557022.804500 punch from 0x02e247 to 0x03e1e5 (0xff9f bytes) 000194 1787557022.809870 write 0x017bce thru 0x01df39 (0x636c bytes) 000195 1787557022.815185 MIRROR_EXTEND to 2 mirrors 000196 1787557023.851305 fallocate 0x032680 thru 0x03dd31 (0xb6b2 bytes) 000197 1787557023.896487 mapread 0x011fad thru 0x01ce9e (0xaef2 bytes) 000198 1787557023.929711 fallocate 0x029cbe thru 0x0387e5 (0xeb28 bytes) 000199 1787557023.937580 trunc from 0x040000 to 0x00d2bc (0x32d43 bytes) 000200 1787557023.965479 trunc from 0x00d2bd to 0x03b0f4 (0x2de38 bytes) 000201 1787557023.985154 trunc from 0x03b0f5 to 0x0364fc (0x4bf8 bytes) 000202 1787557024.006320 mapwrite 0x0015b0 thru 0x003c2c (0x267d bytes) 000203 1787557024.031288 mapwrite 0x01650f thru 0x021031 (0xab23 bytes) 000204 1787557024.073397 trunc from 0x0364fd to 0x020901 (0x15bfb bytes) 000205 1787557024.096594 mapread 0x003abb thru 0x011390 (0xd8d6 bytes) 000206 1787557024.097922 mapread 0x0164a9 thru 0x020901 (0xa459 bytes) 000207 1787557024.099553 fallocate 0x004432 thru 0x00c2d5 (0x7ea4 bytes) 000208 1787557024.105342 trunc from 0x020902 to 0x00ceb7 (0x13a4a bytes) 000210 1787557024.129506 trunc from 0x007a0b to 0x00ceb7 (0x54ad bytes) 000213 1787557024.137674 mapwrite 0x03be65 thru 0x03ffff (0x419b bytes) 000214 1787557024.187943 mapwrite 0x037741 thru 0x038687 (0xf47 bytes) 000215 1787557024.198920 mapwrite 0x037a6f thru 0x039922 (0x1eb4 bytes) 000216 1787557024.210844 trunc from 0x040000 to 0x028c84 (0x1737b bytes) 000217 1787557024.231208 read 0x0073f0 thru 0x01410b (0xcd1c bytes) 000218 1787557024.241810 mapread 0x001006 thru 0x00f088 (0xe083 bytes) 000219 1787557024.243206 read 0x025767 thru 0x027cb9 (0x2553 bytes) 000220 1787557024.259170 write 0x035fc0 thru 0x03ffff (0xa040 bytes) 000221 1787557024.268236 MIRROR_RESYNC 2 mirrors 000222 1787557025.007380 MIRROR_EXTEND to 3 mirrors 000223 1787557025.816434 trunc from 0x014f77 to 0x022cf5 (0xdd7f bytes) 000224 1787557025.848495 trunc from 0x040000 to 0x00c70c (0x338f3 bytes) 000225 1787557025.890085 fallocate 0x03ba0b thru 0x03ffff (0x45f5 bytes) 000226 1787557025.904448 mapread 0x00c2ce thru 0x00c70c (0x43f bytes) 000227 1787557025.915144 mapwrite 0x002466 thru 0x004a71 (0x260c bytes) 000228 1787557025.960467 fallocate 0x01788e thru 0x0277c2 (0xff35 bytes) 000229 1787557025.975338 fallocate 0x005b48 thru 0x01150b (0xb9c4 bytes) 000230 1787557025.985906 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc70d): Mirror 3: 0x7068151f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc70d): Mirror 3: 0x7068151f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc70d: skipped 000231 1787557026.420970 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xc70d): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc70d): Mirror 3: 0x7068151f (stale) CRC-32 checksum value for chunk [0, 0xc70d): Mirror 1: 0x4a75559a Mirror 4: 0x4a75559a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc70d: skipped 000232 1787557027.394945 trunc from 0x00c70d to 0x005ba6 (0x6b66 bytes) 000233 1787557027.455776 fallocate 0x005c90 thru 0x00cb7b (0x6eec bytes) 000234 1787557027.465292 mapread 0x00be6f thru 0x00cb7b (0xd0d bytes) 000235 1787557027.478933 mapwrite 0x03ae0f thru 0x03ffff (0x51f1 bytes) 000237 1787557027.563525 write 0x02175c thru 0x0226fd (0xfa2 bytes) 000239 1787557027.583072 write 0x013a32 thru 0x013dbf (0x38e bytes) 000240 1787557027.596938 mapwrite 0x01de8e thru 0x022a5b (0x4bce bytes) 000241 1787557027.646590 write 0x0347e8 thru 0x03f857 (0xb070 bytes) 000242 1787557027.657758 MIRROR_RESYNC 3 mirrors 000243 1787557028.671407 mapwrite 0x01badc thru 0x023c1b (0x8140 bytes) 000244 1787557028.807852 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8cfe5ef1 (stale) Mirror 4: 0x8cfe5ef1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x26b2ce50 Mirror 5: 0x26b2ce50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000245 1787557029.862917 fallocate 0x0296f9 thru 0x035253 (0xbb5b bytes) 000246 1787557029.908317 mapwrite 0x00de94 thru 0x017ab6 (0x9c23 bytes) 000247 1787557029.988835 fallocate 0x001857 thru 0x00c921 (0xb0cb bytes) 000248 1787557030.003146 write 0x00b68a thru 0x00dd61 (0x26d8 bytes) 000250 1787557030.017545 mapwrite 0x01885c thru 0x0287a3 (0xff48 bytes) 000251 1787557030.101928 read 0x019073 thru 0x01da83 (0x4a11 bytes) 000252 1787557030.103080 mapwrite 0x006870 thru 0x00e6ff (0x7e90 bytes) 000253 1787557030.147378 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8cfe5ef1 (stale) Mirror 5: 0x26b2ce50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8cfe5ef1 (stale) Mirror 5: 0x26b2ce50 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000254 1787557030.537388 write 0x01a97d thru 0x01c099 (0x171d bytes) 000255 1787557030.554753 punch from 0x00c99b to 0x018db4 (0xc41a bytes) 000256 1787557030.565512 read 0x00bf90 thru 0x01825a (0xc2cb bytes) 000257 1787557030.575062 trunc from 0x040000 to 0x01bb62 (0x2449d bytes) 000258 1787557030.600190 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1bb63): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bb63): Mirror 4: 0x1322e362 (stale) Mirror 5: 0x6015566f (stale) CRC-32 checksum value for chunk [0, 0x1bb63): Mirror 1: 0x638a8d88 Mirror 6: 0x638a8d88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bb63: skipped 000259 1787557031.564559 MIRROR_RESYNC 4 mirrors 000260 1787557032.345376 punch from 0x006627 to 0x011a65 (0xb43f bytes) 000261 1787557032.379223 trunc from 0x0106e1 to 0x01bb62 (0xb482 bytes) 000263 1787557032.385464 trunc from 0x01bb63 to 0x029852 (0xdcf0 bytes) 000264 1787557032.406183 read 0x00061a thru 0x00c8bc (0xc2a3 bytes) 000265 1787557032.418030 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29853): Mirror 4: 0x3c1172c2 (stale) Mirror 6: 0x3c1172c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29853): Mirror 4: 0x3c1172c2 (stale) Mirror 6: 0x3c1172c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29853: skipped 000266 1787557032.831963 MIRROR_RESYNC 3 mirrors 000267 1787557033.787716 trunc from 0x029853 to 0x00b803 (0x1e04f bytes) 000268 1787557033.834183 trunc from 0x0088a7 to 0x00b803 (0x2f5d bytes) 000270 1787557033.839903 fallocate 0x00d8f6 thru 0x0186d8 (0xade3 bytes) 000271 1787557033.846263 trunc from 0x008e9e to 0x00b803 (0x2966 bytes) 000272 1787557033.850401 read 0x0040cb thru 0x00852b (0x4461 bytes) 000273 1787557033.859261 write 0x02cdf5 thru 0x036bed (0x9df9 bytes) 000274 1787557033.860750 mapread 0x01a5d7 thru 0x01b202 (0xc2c bytes) 000275 1787557033.869466 trunc from 0x036bee to 0x01604e (0x20b9f bytes) 000276 1787557033.886593 fallocate 0x003d62 thru 0x00b4b4 (0x7753 bytes) 000277 1787557033.891946 write 0x0045ab thru 0x012328 (0xdd7e bytes) 000279 1787557033.907228 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1604f): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1604f): Mirror 4: 0xf83a4d89 (stale) Mirror 6: 0xf83a4d89 (stale) CRC-32 checksum value for chunk [0, 0x1604f): Mirror 1: 0x195bdad3 Mirror 7: 0x195bdad3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1604f: skipped 000280 1787557035.014315 mapwrite 0x023bfa thru 0x031e6a (0xe271 bytes) 000281 1787557035.162882 punch from 0x0231fb to 0x0261a1 (0x2fa7 bytes) 000282 1787557035.174415 read 0x00ffb4 thru 0x013219 (0x3266 bytes) 000283 1787557035.191715 fallocate 0x02a314 thru 0x0375b3 (0xd2a0 bytes) 000284 1787557035.198407 trunc from 0x01a8c2 to 0x0229b5 (0x80f4 bytes) 000285 1787557035.209207 trunc from 0x0375b4 to 0x015e56 (0x2175d bytes) 000286 1787557035.241502 read 0x005640 thru 0x014cdb (0xf69c bytes) 000287 1787557035.271163 fallocate 0x031a53 thru 0x034f6e (0x351c bytes) 000288 1787557035.282801 fallocate 0x0161f6 thru 0x021881 (0xb68c bytes) 000289 1787557035.293896 punch from 0x009c98 to 0x014d0e (0xb077 bytes) 000290 1787557035.306590 trunc from 0x034f6f to 0x0279a1 (0xd5cd bytes) 000291 1787557035.339354 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x279a2): Mirror 6: 0x7cae9667 (stale) Mirror 7: 0x7873bd8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x279a2): Mirror 6: 0x7cae9667 (stale) Mirror 7: 0x7873bd8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x279a2: skipped 000292 1787557035.818766 punch from 0x009b52 to 0x019a0c (0xfebb bytes) 000293 1787557035.823468 trunc from 0x0279a2 to 0x03a732 (0x12d91 bytes) 000295 1787557035.882569 read 0x03a024 thru 0x03a732 (0x70f bytes) 000296 1787557035.905217 mapwrite 0x0176ee thru 0x020685 (0x8f98 bytes) 000297 1787557035.996042 trunc from 0x03a733 to 0x0235b6 (0x1717c bytes) 000298 1787557036.024885 trunc from 0x016f9d to 0x01f0b8 (0x811c bytes) 000299 1787557036.035129 mapwrite 0x015c42 thru 0x01a20b (0x45ca bytes) 000300 1787557036.099522 fallocate 0x02ef38 thru 0x03449a (0x5563 bytes) 000301 1787557036.111754 trunc from 0x03449b to 0x0170d9 (0x1d3c1 bytes) 000302 1787557036.149111 read 0x015275 thru 0x0170d9 (0x1e65 bytes) 000303 1787557036.156193 MIRROR_RESYNC 3 mirrors 000304 1787557037.075927 trunc from 0x0170da to 0x007c22 (0xf4b7 bytes) 000305 1787557037.214420 mapwrite 0x013cda thru 0x01d68b (0x99b2 bytes) 000306 1787557037.338373 trunc from 0x01d68c to 0x005877 (0x17e14 bytes) 000307 1787557037.371105 mapread 0x0023ca thru 0x005877 (0x34ae bytes) 000308 1787557037.385019 read 0x003db1 thru 0x005877 (0x1ac7 bytes) 000309 1787557037.390159 MIRROR_RESYNC 3 mirrors 000310 1787557037.959411 mapread 0x002e63 thru 0x00583e (0x29dc bytes) 000311 1787557037.973898 read 0x001276 thru 0x005877 (0x4602 bytes) 000312 1787557037.974277 mapwrite 0x00b65b thru 0x01a41e (0xedc4 bytes) 000313 1787557038.177329 fallocate 0x02f3d9 thru 0x03d0f3 (0xdd1b bytes) 000314 1787557038.187013 read 0x01dcdc thru 0x02cb81 (0xeea6 bytes) 000315 1787557038.206345 read 0x02517f thru 0x027158 (0x1fda bytes) 000316 1787557038.207401 trunc from 0x032862 to 0x03d0f3 (0xa892 bytes) 000317 1787557038.219462 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d0f4): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0f4): Mirror 6: 0x353cf7dc (stale) Mirror 7: 0x353cf7dc (stale) CRC-32 checksum value for chunk [0, 0x3d0f4): Mirror 1: 0xc79ec9be Mirror 8: 0xc79ec9be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0f4: skipped 000318 1787557039.088158 read 0x0218ce thru 0x02b470 (0x9ba3 bytes) 000319 1787557039.112433 read 0x014ec0 thru 0x0178ea (0x2a2b bytes) 000320 1787557039.153898 read 0x03c082 thru 0x03d0f3 (0x1072 bytes) 000321 1787557039.172465 trunc from 0x0107d2 to 0x020476 (0xfca5 bytes) 000322 1787557039.261946 fallocate 0x02ae37 thru 0x02f9e8 (0x4bb2 bytes) 000323 1787557039.268287 write 0x013ffa thru 0x01d0ed (0x90f4 bytes) 000324 1787557039.336435 write 0x01cf80 thru 0x01db6a (0xbeb bytes) 000326 1787557039.337879 mapwrite 0x023137 thru 0x03286a (0xf734 bytes) 000327 1787557039.426564 trunc from 0x03d0f4 to 0x016608 (0x26aeb bytes) 000328 1787557039.532523 MIRROR_RESYNC 4 mirrors 000329 1787557040.864169 MIRROR_RESYNC 4 mirrors 000330 1787557041.288758 fallocate 0x03dff8 thru 0x03ffff (0x2008 bytes) 000331 1787557041.350000 mapread 0x009908 thru 0x00ccab (0x33a4 bytes) 000332 1787557041.381123 trunc from 0x016609 to 0x023744 (0xd13c bytes) 000333 1787557041.420326 write 0x02b1ba thru 0x035a37 (0xa87e bytes) 000334 1787557041.425998 mapread 0x015c3c thru 0x016cf3 (0x10b8 bytes) 000335 1787557041.448499 mapwrite 0x030d4e thru 0x03eb6b (0xde1e bytes) 000337 1787557041.529189 MIRROR_RESYNC 4 mirrors 000338 1787557042.840291 read 0x005e2a thru 0x00b2c3 (0x549a bytes) 000339 1787557042.849863 mapwrite 0x00fa5e thru 0x01e9b2 (0xef55 bytes) 000340 1787557043.028853 mapwrite 0x0380f5 thru 0x03ffff (0x7f0b bytes) 000341 1787557043.113485 mapwrite 0x00cc0f thru 0x00e664 (0x1a56 bytes) 000342 1787557043.135118 fallocate 0x005e19 thru 0x007bc4 (0x1dac bytes) 000343 1787557043.142483 mapread 0x00d6f2 thru 0x01c3cd (0xecdc bytes) 000346 1787557043.148075 mapread 0x03cf2e thru 0x03ffff (0x30d2 bytes) 000347 1787557043.149986 mapwrite 0x031193 thru 0x036cb5 (0x5b23 bytes) 000348 1787557043.167476 mapread 0x021d71 thru 0x02d047 (0xb2d7 bytes) 000349 1787557043.183209 read 0x008e94 thru 0x015fff (0xd16c bytes) 000351 1787557043.190580 write 0x00ce6f thru 0x01039c (0x352e bytes) 000352 1787557043.191741 punch from 0x02e24e to 0x0346d7 (0x648a bytes) 000354 1787557043.197432 mapread 0x02421a thru 0x029941 (0x5728 bytes) 000355 1787557043.199226 trunc from 0x040000 to 0x01ccc6 (0x23339 bytes) 000356 1787557043.232813 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ccc7): Mirror 6: 0xee12708 (stale) Mirror 7: 0xee12708 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ccc7): Mirror 6: 0xee12708 (stale) Mirror 7: 0xee12708 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ccc7: skipped 000357 1787557043.705450 mapwrite 0x01fad2 thru 0x02e800 (0xed2f bytes) 000358 1787557043.830264 trunc from 0x02e14c to 0x02e800 (0x6b5 bytes) 000360 1787557043.842529 trunc from 0x01a325 to 0x01ed89 (0x4a65 bytes) 000362 1787557043.861429 read 0x001ef9 thru 0x004f81 (0x3089 bytes) 000363 1787557043.892917 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e801): Mirror 7: 0x8149f802 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e801): Mirror 7: 0x8149f802 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e801: skipped 000366 1787557044.294334 trunc from 0x013472 to 0x021768 (0xe2f7 bytes) 000367 1787557044.299912 trunc from 0x02e801 to 0x005479 (0x29387 bytes) 000369 1787557044.334313 read 0x002954 thru 0x005479 (0x2b26 bytes) 000370 1787557044.344378 write 0x039a0a thru 0x03bc3d (0x2234 bytes) 000371 1787557044.346823 punch from 0x017afb to 0x01d35b (0x5861 bytes) 000372 1787557044.355219 MIRROR_RESYNC 2 mirrors 000373 1787557044.823483 punch from 0x03026a to 0x037f9f (0x7d36 bytes) 000374 1787557044.858821 mapread 0x01887f thru 0x0229df (0xa161 bytes) 000375 1787557044.881352 mapread 0x00ae01 thru 0x01054d (0x574d bytes) 000376 1787557044.890326 write 0x00c5cc thru 0x00c7fd (0x232 bytes) 000377 1787557044.893644 fallocate 0x00feea thru 0x01ad14 (0xae2b bytes) 000378 1787557044.900542 read 0x022ce5 thru 0x0329e3 (0xfcff bytes) 000379 1787557044.919504 mapwrite 0x00d741 thru 0x01c49c (0xed5c bytes) 000380 1787557044.943092 punch from 0x02b37f to 0x0323a0 (0x7022 bytes) 000381 1787557044.948542 fallocate 0x03d705 thru 0x03ffff (0x28fb bytes) 000382 1787557044.954967 mapread 0x022d38 thru 0x031e0b (0xf0d4 bytes) 000383 1787557044.967968 punch from 0x003bc4 to 0x00d8a2 (0x9cdf bytes) 000384 1787557044.984173 punch from 0x02f184 to 0x0301e9 (0x1066 bytes) 000385 1787557044.995481 trunc from 0x00077e to 0x00b33a (0xabbd bytes) 000386 1787557045.003721 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) 000387 1787557045.221474 write 0x02b7b3 thru 0x0326fd (0x6f4b bytes) 000388 1787557045.252507 fallocate 0x03272f thru 0x035c86 (0x3558 bytes) 000389 1787557045.264162 punch from 0x00fa89 to 0x01be8e (0xc406 bytes) 000390 1787557045.280339 trunc from 0x03afae to 0x03bc3d (0xc90 bytes) 000391 1787557045.287946 mapwrite 0x028ca5 thru 0x034483 (0xb7df bytes) 000392 1787557045.356827 write 0x009d64 thru 0x015d92 (0xc02f bytes) 000394 1787557045.376503 mapread 0x02dc74 thru 0x02f010 (0x139d bytes) 000396 1787557045.383304 mapwrite 0x007b77 thru 0x017980 (0xfe0a bytes) 000398 1787557045.469792 read 0x039784 thru 0x03bc3d (0x24ba bytes) 000399 1787557045.471150 trunc from 0x02d51f to 0x03bc3d (0xe71f bytes) 000400 1787557045.483118 punch from 0x0394e7 to 0x03bc3d (0x2757 bytes) 000401 1787557045.493031 fallocate 0x02381b thru 0x023f38 (0x71e bytes) 000402 1787557045.502127 mapread 0x03550a thru 0x037493 (0x1f8a bytes) 000403 1787557045.520067 fallocate 0x00bbb0 thru 0x00de56 (0x22a7 bytes) 000404 1787557045.524776 mapwrite 0x005a3e thru 0x007267 (0x182a bytes) 000405 1787557045.537046 read 0x0386c7 thru 0x03bc3d (0x3577 bytes) 000406 1787557045.538042 read 0x018288 thru 0x01ccca (0x4a43 bytes) 000407 1787557045.538669 fallocate 0x036834 thru 0x03ffff (0x97cc bytes) 000408 1787557045.547152 MIRROR_EXTEND to 2 mirrors 000409 1787557046.615361 trunc from 0x00a82e to 0x0125c2 (0x7d95 bytes) 000410 1787557046.663258 read 0x03b4e2 thru 0x03bc3d (0x75c bytes) 000411 1787557046.670383 read 0x028e7f thru 0x032443 (0x95c5 bytes) 000412 1787557046.686697 trunc from 0x03bc3e to 0x01fd3c (0x1bf01 bytes) 000413 1787557046.721962 punch from 0x01b0d8 to 0x01df8b (0x2eb4 bytes) 000414 1787557046.728772 mapwrite 0x02e31a thru 0x03cc12 (0xe8f9 bytes) 000415 1787557046.941587 fallocate 0x03c570 thru 0x03ffff (0x3a90 bytes) 000416 1787557046.955354 punch from 0x0383e8 to 0x03ffff (0x7c18 bytes) 000417 1787557046.988553 punch from 0x02644e to 0x02b65b (0x520e bytes) 000418 1787557047.003492 write 0x0006a7 thru 0x001c8a (0x15e4 bytes) 000419 1787557047.012009 mapread 0x010f77 thru 0x0129c3 (0x1a4d bytes) 000421 1787557047.014387 trunc from 0x040000 to 0x03a8bf (0x5740 bytes) 000422 1787557047.055344 trunc from 0x03a8c0 to 0x01cc10 (0x1dcaf bytes) 000423 1787557047.087202 write 0x039f7a thru 0x03f73c (0x57c3 bytes) 000424 1787557047.095295 fallocate 0x00193a thru 0x0034d7 (0x1b9e bytes) 000425 1787557047.101264 write 0x00b7b3 thru 0x01998a (0xe1d8 bytes) 000427 1787557047.108547 trunc from 0x019edf to 0x01fe3b (0x5f5d bytes) 000428 1787557047.127036 mapwrite 0x004fd9 thru 0x011164 (0xc18c bytes) 000429 1787557047.156070 mapwrite 0x03eeb9 thru 0x03ffff (0x1147 bytes) 000430 1787557047.232079 punch from 0x00f8d7 to 0x01d1a6 (0xd8d0 bytes) 000431 1787557047.252557 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1eb53d9c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x719282be Mirror 3: 0x719282be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000432 1787557048.086710 fallocate 0x017899 thru 0x01f7e8 (0x7f50 bytes) 000433 1787557048.125509 punch from 0x039194 to 0x03d51b (0x4388 bytes) 000434 1787557048.133113 trunc from 0x016770 to 0x017c91 (0x1522 bytes) 000435 1787557048.138356 trunc from 0x01f7ca to 0x029f21 (0xa758 bytes) 000436 1787557048.147077 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x719282be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x719282be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000437 1787557048.464279 trunc from 0x040000 to 0x033004 (0xcffb bytes) 000438 1787557048.576092 write 0x01aafb thru 0x01b376 (0x87c bytes) 000439 1787557048.584491 trunc from 0x033005 to 0x025add (0xd527 bytes) 000440 1787557048.597517 trunc from 0x025ade to 0x025159 (0x984 bytes) 000441 1787557048.611282 punch from 0x00a315 to 0x0108f0 (0x65dc bytes) 000442 1787557048.616432 write 0x02e5fa thru 0x031ee1 (0x38e8 bytes) 000443 1787557048.617343 trunc from 0x031ee2 to 0x022403 (0xfade bytes) 000445 1787557048.633681 trunc from 0x022404 to 0x010372 (0x12091 bytes) 000446 1787557048.646298 read 0x00e429 thru 0x010372 (0x1f4a bytes) 000448 1787557048.655926 trunc from 0x00f6ab to 0x010372 (0xcc8 bytes) 000449 1787557048.660394 trunc from 0x005cb7 to 0x010372 (0xa6bc bytes) 000450 1787557048.667315 read 0x00b02a thru 0x010372 (0x5349 bytes) 000451 1787557048.684083 read 0x009fa6 thru 0x010372 (0x63cd bytes) 000453 1787557048.685291 mapwrite 0x0051a7 thru 0x0060f9 (0xf53 bytes) 000454 1787557048.725184 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) 000455 1787557048.916778 fallocate 0x0155d2 thru 0x01cf19 (0x7948 bytes) 000456 1787557048.935967 mapwrite 0x00ec28 thru 0x012233 (0x360c bytes) 000458 1787557049.061322 trunc from 0x012234 to 0x009a34 (0x87ff bytes) 000459 1787557049.086919 MIRROR_EXTEND to 2 mirrors 000461 1787557049.937953 mapread 0x0003a1 thru 0x007812 (0x7472 bytes) 000462 1787557049.972628 mapread 0x0071c4 thru 0x009a34 (0x2871 bytes) 000463 1787557049.985905 MIRROR_EXTEND to 3 mirrors 000464 1787557051.036176 fallocate 0x01d9f1 thru 0x025d43 (0x8353 bytes) 000466 1787557051.096275 fallocate 0x03813c thru 0x03ffff (0x7ec4 bytes) 000467 1787557051.107978 fallocate 0x009ed3 thru 0x0119da (0x7b08 bytes) 000468 1787557051.116962 punch from 0x01fc45 to 0x0299f2 (0x9dae bytes) 000469 1787557051.123015 read 0x00e4f2 thru 0x012af8 (0x4607 bytes) 000470 1787557051.133310 read 0x03af86 thru 0x03d54f (0x25ca bytes) 000472 1787557051.157350 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfbe97229 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfbe97229 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000473 1787557051.567131 punch from 0x016c41 to 0x018857 (0x1c17 bytes) 000474 1787557051.575496 read 0x013dab thru 0x01fb89 (0xbddf bytes) 000475 1787557051.589716 mapread 0x023d68 thru 0x03089f (0xcb38 bytes) 000476 1787557051.617028 mapread 0x01ead1 thru 0x01fb56 (0x1086 bytes) 000477 1787557051.617996 trunc from 0x01ca20 to 0x027cd3 (0xb2b4 bytes) 000478 1787557051.623223 mapread 0x02e797 thru 0x0332db (0x4b45 bytes) 000479 1787557051.628078 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfbe97229 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfbe97229 Mirror 4: 0xfbe97229 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000480 1787557053.190008 mapwrite 0x00b930 thru 0x016aed (0xb1be bytes) 000481 1787557053.394047 trunc from 0x00fd1e to 0x01c873 (0xcb56 bytes) 000482 1787557053.428177 mapread 0x024113 thru 0x02fe07 (0xbcf5 bytes) 000483 1787557053.490051 read 0x02f73c thru 0x037e34 (0x86f9 bytes) 000484 1787557053.500057 trunc from 0x040000 to 0x0219ed (0x1e612 bytes) 000485 1787557053.537090 trunc from 0x0219ee to 0x01d1d5 (0x4818 bytes) 000486 1787557053.575426 mapread 0x00ca0e thru 0x01b955 (0xef48 bytes) 000487 1787557053.608798 trunc from 0x01d1d6 to 0x007e08 (0x153cd bytes) 000488 1787557053.632117 write 0x03400b thru 0x03ffff (0xbff5 bytes) 000489 1787557053.636981 punch from 0x00d2d6 to 0x01756c (0xa297 bytes) 000490 1787557053.646260 write 0x014b2c thru 0x02498b (0xfe60 bytes) 000492 1787557053.673088 trunc from 0x022802 to 0x0261b0 (0x39af bytes) 000493 1787557053.726994 trunc from 0x01a85d to 0x02a672 (0xfe16 bytes) 000494 1787557053.733818 fallocate 0x02be5a thru 0x035398 (0x953f bytes) 000495 1787557053.759891 write 0x02eacc thru 0x03622c (0x7761 bytes) 000496 1787557053.782952 fallocate 0x0248b6 thru 0x0257d9 (0xf24 bytes) 000498 1787557053.804837 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfbe97229 (stale) Mirror 4: 0xfbe97229 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb8599373 Mirror 5: 0xb8599373 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000499 1787557055.589833 write 0x0079e6 thru 0x016fdd (0xf5f8 bytes) 000500 1787557055.699800 trunc from 0x02ef63 to 0x03db11 (0xebaf bytes) 000501 1787557055.711345 mapread 0x01f58c thru 0x02ae6a (0xb8df bytes) 000503 1787557055.772435 trunc from 0x040000 to 0x0007c6 (0x3f839 bytes) 000504 1787557055.850980 trunc from 0x0007c7 to 0x00dc01 (0xd43b bytes) 000505 1787557055.873096 punch from 0x006043 to 0x00c9a3 (0x6961 bytes) 000506 1787557055.884071 MIRROR_RESYNC 4 mirrors 000507 1787557056.888082 write 0x037d58 thru 0x03adf6 (0x309f bytes) 000508 1787557056.929048 trunc from 0x0273d8 to 0x036a0a (0xf633 bytes) 000509 1787557056.942003 trunc from 0x03adf7 to 0x03dcce (0x2ed8 bytes) 000510 1787557056.961084 mapwrite 0x02620d thru 0x033979 (0xd76d bytes) 000511 1787557057.048228 read 0x006c5e thru 0x010507 (0x98aa bytes) 000512 1787557057.060883 trunc from 0x03dccf to 0x00dfed (0x2fce1 bytes) 000513 1787557057.118863 trunc from 0x0005d1 to 0x006431 (0x5e61 bytes) 000514 1787557057.142448 mapwrite 0x034ab6 thru 0x03f086 (0xa5d1 bytes) 000515 1787557057.248220 mapread 0x025e79 thru 0x032520 (0xc6a8 bytes) 000516 1787557057.260107 write 0x01c0de thru 0x02231a (0x623d bytes) 000517 1787557057.267688 mapwrite 0x0336a5 thru 0x03733c (0x3c98 bytes) 000518 1787557057.309542 fallocate 0x01d070 thru 0x021682 (0x4613 bytes) 000521 1787557057.331784 MIRROR_RESYNC 4 mirrors 000522 1787557058.329385 punch from 0x02aab9 to 0x02d5ea (0x2b32 bytes) 000523 1787557058.363402 punch from 0x01da49 to 0x028603 (0xabbb bytes) 000524 1787557058.368993 read 0x03abd6 thru 0x03df32 (0x335d bytes) 000525 1787557058.375303 mapwrite 0x02794e thru 0x030820 (0x8ed3 bytes) 000526 1787557058.427076 trunc from 0x022397 to 0x031e2c (0xfa96 bytes) 000527 1787557058.436689 read 0x0211d1 thru 0x02b109 (0x9f39 bytes) 000528 1787557058.453301 fallocate 0x001367 thru 0x00a591 (0x922b bytes) 000530 1787557058.465909 read 0x021921 thru 0x024c53 (0x3333 bytes) 000531 1787557058.466563 fallocate 0x032783 thru 0x03cc84 (0xa502 bytes) 000532 1787557058.475583 read 0x03abf9 thru 0x03d38b (0x2793 bytes) 000534 1787557058.496156 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f087): Mirror 4: 0x7c3c3bd (stale) Mirror 5: 0x7c3c3bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f087): Mirror 4: 0x7c3c3bd (stale) Mirror 5: 0x7c3c3bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f087: skipped 000535 1787557058.877175 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f087): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f087): Mirror 4: 0x7c3c3bd (stale) Mirror 5: 0x7c3c3bd (stale) CRC-32 checksum value for chunk [0, 0x3f087): Mirror 1: 0x9afccd64 Mirror 6: 0x9afccd64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f087: skipped 000536 1787557060.426044 trunc from 0x03f087 to 0x01f839 (0x1f84d bytes) 000537 1787557060.566997 mapwrite 0x03244b thru 0x037dde (0x5994 bytes) 000538 1787557060.783841 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37ddf): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ddf): Mirror 4: 0xad3204 (stale) Mirror 5: 0xad3204 (stale) Mirror 6: 0x3581f117 (stale) CRC-32 checksum value for chunk [0, 0x37ddf): Mirror 1: 0xde5f62f5 Mirror 7: 0xde5f62f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ddf: skipped 000539 1787557062.570222 trunc from 0x032c2c to 0x035da4 (0x3179 bytes) 000540 1787557062.646497 mapwrite 0x007661 thru 0x00e837 (0x71d7 bytes) 000541 1787557062.750011 write 0x037acb thru 0x03ffff (0x8535 bytes) 000543 1787557062.780935 punch from 0x0351cd to 0x03ffff (0xae33 bytes) 000544 1787557062.840459 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x22797b9e (stale) Mirror 5: 0x22797b9e (stale) Mirror 6: 0xb6e877a0 (stale) Mirror 7: 0x892fd350 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x37e6ef3f Mirror 8: 0x37e6ef3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000545 1787557064.475995 mapread 0x006549 thru 0x013b4e (0xd606 bytes) 000546 1787557064.520955 trunc from 0x0259ae to 0x027973 (0x1fc6 bytes) 000548 1787557064.561557 punch from 0x0292f5 to 0x0334c3 (0xa1cf bytes) 000549 1787557064.566376 write 0x01f5d2 thru 0x02bc97 (0xc6c6 bytes) 000551 1787557064.587103 read 0x028bab thru 0x035004 (0xc45a bytes) 000552 1787557064.607173 mapread 0x013f7c thru 0x01f875 (0xb8fa bytes) 000553 1787557064.655788 mapread 0x02375f thru 0x03318a (0xfa2c bytes) 000554 1787557064.656746 write 0x03cdd3 thru 0x03f86e (0x2a9c bytes) 000555 1787557064.682755 punch from 0x020d76 to 0x0259e1 (0x4c6c bytes) 000556 1787557064.728425 read 0x00a60f thru 0x00fc96 (0x5688 bytes) 000557 1787557064.743224 write 0x011140 thru 0x01a668 (0x9529 bytes) 000558 1787557064.745161 trunc from 0x02ae7a to 0x036808 (0xb98f bytes) 000559 1787557064.753537 mapread 0x0167e6 thru 0x01dab9 (0x72d4 bytes) 000560 1787557064.755327 mapread 0x01ac8d thru 0x01ca4e (0x1dc2 bytes) 000561 1787557064.756977 fallocate 0x006f59 thru 0x00cccb (0x5d73 bytes) 000562 1787557064.763385 punch from 0x011070 to 0x01eb40 (0xdad1 bytes) 000563 1787557064.778015 mapwrite 0x018d1c thru 0x01f197 (0x647c bytes) 000564 1787557064.851293 punch from 0x005033 to 0x0136d6 (0xe6a4 bytes) 000565 1787557064.859996 punch from 0x01779d to 0x021515 (0x9d79 bytes) 000566 1787557064.867586 trunc from 0x040000 to 0x032c46 (0xd3b9 bytes) 000567 1787557064.905545 mapread 0x0108ce thru 0x01258f (0x1cc2 bytes) 000568 1787557064.936811 trunc from 0x032c47 to 0x00caef (0x26157 bytes) 000569 1787557064.963029 fallocate 0x00cd46 thru 0x012f8b (0x6246 bytes) 000570 1787557064.979062 mapwrite 0x01f335 thru 0x023ac8 (0x4794 bytes) 000571 1787557065.077726 write 0x03f0bf thru 0x03ffff (0xf41 bytes) 000572 1787557065.082985 trunc from 0x03a7f3 to 0x03ffff (0x580d bytes) 000574 1787557065.107212 trunc from 0x02249a to 0x026e99 (0x4a00 bytes) 000575 1787557065.125134 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x22797b9e (stale) Mirror 5: 0x22797b9e (stale) Mirror 6: 0xb6e877a0 (stale) Mirror 7: 0x892fd350 (stale) Mirror 8: 0x37e6ef3f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5d0c75cc Mirror 9: 0x5d0c75cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000576 1787557066.554634 fallocate 0x01f634 thru 0x026592 (0x6f5f bytes) 000577 1787557066.624718 fallocate 0x03cffb thru 0x03ea6e (0x1a74 bytes) 000578 1787557066.635018 trunc from 0x01d7c5 to 0x029402 (0xbc3e bytes) 000581 1787557066.658795 mapread 0x02bf22 thru 0x0394a9 (0xd588 bytes) 000582 1787557066.711411 fallocate 0x014a5b thru 0x02384c (0xedf2 bytes) 000583 1787557066.723431 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x22797b9e (stale) Mirror 6: 0xb6e877a0 (stale) Mirror 7: 0x892fd350 (stale) Mirror 8: 0x37e6ef3f (stale) Mirror 9: 0x5d0c75cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x22797b9e (stale) Mirror 6: 0xb6e877a0 (stale) Mirror 7: 0x892fd350 (stale) Mirror 8: 0x37e6ef3f (stale) Mirror 9: 0x5d0c75cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000584 1787557067.301816 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb6e877a0 (stale) Mirror 7: 0x892fd350 (stale) Mirror 8: 0x37e6ef3f (stale) Mirror 9: 0x5d0c75cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xb6e877a0 (stale) Mirror 7: 0x892fd350 (stale) Mirror 8: 0x37e6ef3f (stale) Mirror 9: 0x5d0c75cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000585 1787557067.835031 punch from 0x023166 to 0x02b031 (0x7ecc bytes) 000586 1787557067.839374 mapwrite 0x028a60 thru 0x02dbee (0x518f bytes) 000587 1787557067.903019 MIRROR_RESYNC 5 mirrors 000588 1787557069.501517 trunc from 0x039546 to 0x03dfa7 (0x4a62 bytes) 000589 1787557069.542557 mapread 0x03684d thru 0x03d9d2 (0x7186 bytes) 000590 1787557069.578286 trunc from 0x040000 to 0x037745 (0x88ba bytes) 000591 1787557069.612091 punch from 0x0201d8 to 0x022910 (0x2739 bytes) 000592 1787557069.617375 trunc from 0x00d9a0 to 0x017a17 (0xa078 bytes) 000593 1787557069.622181 fallocate 0x022e40 thru 0x02f6f4 (0xc8b5 bytes) 000594 1787557069.628549 write 0x02a922 thru 0x03506e (0xa74d bytes) 000595 1787557069.641390 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37746): Mirror 6: 0xddaaf7ce (stale) Mirror 7: 0xddaaf7ce (stale) Mirror 9: 0xddaaf7ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37746): Mirror 6: 0xddaaf7ce (stale) Mirror 7: 0xddaaf7ce (stale) Mirror 9: 0xddaaf7ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37746: skipped 000597 1787557070.099854 mapwrite 0x00d819 thru 0x017cd3 (0xa4bb bytes) 000598 1787557070.145151 fallocate 0x00c08b thru 0x016885 (0xa7fb bytes) 000599 1787557070.152432 trunc from 0x00b024 to 0x01800b (0xcfe8 bytes) 000601 1787557070.159900 write 0x036c46 thru 0x03ada1 (0x415c bytes) 000602 1787557070.173786 mapread 0x023ee3 thru 0x0282f8 (0x4416 bytes) 000603 1787557070.186206 trunc from 0x03ada2 to 0x01e085 (0x1cd1c bytes) 000604 1787557070.216079 mapread 0x016b0d thru 0x01e085 (0x7579 bytes) 000605 1787557070.234834 punch from 0x008b5a to 0x0117f9 (0x8ca0 bytes) 000606 1787557070.241726 write 0x01f233 thru 0x024071 (0x4e3f bytes) 000607 1787557070.245005 read 0x0109e1 thru 0x0171f7 (0x6817 bytes) 000608 1787557070.250989 trunc from 0x00fb73 to 0x01c441 (0xc8cf bytes) 000609 1787557070.259216 fallocate 0x018e09 thru 0x023e47 (0xb03f bytes) 000611 1787557070.268710 trunc from 0x024072 to 0x023e82 (0x1ef bytes) 000612 1787557070.292388 mapwrite 0x004f73 thru 0x00f4fd (0xa58b bytes) 000613 1787557070.350113 write 0x01afa8 thru 0x025836 (0xa88f bytes) 000614 1787557070.363206 trunc from 0x025837 to 0x005e7a (0x1f9bc bytes) 000615 1787557070.388663 fallocate 0x0211be thru 0x028248 (0x708b bytes) 000616 1787557070.403342 MIRROR_RESYNC 4 mirrors 000617 1787557071.587247 mapread 0x02243a thru 0x024e4c (0x2a13 bytes) 000618 1787557071.607535 mapread 0x0073ea thru 0x007d4d (0x964 bytes) 000619 1787557071.634237 punch from 0x017030 to 0x01b8d6 (0x48a7 bytes) 000620 1787557071.684724 punch from 0x00c6d7 to 0x00d71d (0x1047 bytes) 000622 1787557071.701929 mapwrite 0x000a9a thru 0x00e691 (0xdbf8 bytes) 000623 1787557071.756966 write 0x011bb7 thru 0x01cb6e (0xafb8 bytes) 000624 1787557071.773078 read 0x019f5b thru 0x021437 (0x74dd bytes) 000625 1787557071.785030 write 0x024006 thru 0x02d7a5 (0x97a0 bytes) 000626 1787557071.793151 punch from 0x0046b7 to 0x01450e (0xfe58 bytes) 000627 1787557071.824541 mapwrite 0x0168b8 thru 0x01b525 (0x4c6e bytes) 000628 1787557071.858741 trunc from 0x02d7a6 to 0x03427d (0x6ad8 bytes) 000630 1787557071.901580 write 0x017992 thru 0x02240d (0xaa7c bytes) 000631 1787557071.920236 read 0x02fcf5 thru 0x03427d (0x4589 bytes) 000632 1787557071.932442 trunc from 0x02ece3 to 0x03427d (0x559b bytes) 000633 1787557071.939011 punch from 0x02076c to 0x0240f5 (0x398a bytes) 000634 1787557071.959247 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3427e): Mirror 7: 0x50c132d0 (stale) Mirror 9: 0x50c132d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3427e): Mirror 7: 0x50c132d0 (stale) Mirror 9: 0x50c132d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3427e: skipped 000635 1787557072.394560 write 0x02e78c thru 0x03241d (0x3c92 bytes) 000636 1787557072.413992 mapread 0x025c57 thru 0x0298fe (0x3ca8 bytes) 000637 1787557072.441285 trunc from 0x00ff4e to 0x01b8c7 (0xb97a bytes) 000638 1787557072.447682 trunc from 0x017ff7 to 0x024162 (0xc16c bytes) 000639 1787557072.453864 fallocate 0x03e431 thru 0x03ffff (0x1bcf bytes) 000640 1787557072.459065 fallocate 0x03e5a8 thru 0x03ffff (0x1a58 bytes) 000641 1787557072.464877 trunc from 0x001899 to 0x002613 (0xd7b bytes) 000642 1787557072.475286 write 0x03333d thru 0x03e13f (0xae03 bytes) 000644 1787557072.493363 trunc from 0x01ec61 to 0x02d5df (0xe97f bytes) 000645 1787557072.502705 trunc from 0x040000 to 0x026047 (0x19fb8 bytes) 000646 1787557072.535555 write 0x022154 thru 0x0260c4 (0x3f71 bytes) 000647 1787557072.547224 trunc from 0x0260c5 to 0x008c26 (0x1d49e bytes) 000648 1787557072.569282 write 0x016587 thru 0x01acfd (0x4777 bytes) 000649 1787557072.572340 trunc from 0x01acfe to 0x02dcc2 (0x12fc5 bytes) 000650 1787557072.591006 punch from 0x0045bf to 0x00dfed (0x9a2f bytes) 000651 1787557072.597339 mapwrite 0x025246 thru 0x02d285 (0x8040 bytes) 000652 1787557072.643862 mapwrite 0x03696f thru 0x0374c4 (0xb56 bytes) 000653 1787557072.695332 fallocate 0x016692 thru 0x022c4c (0xc5bb bytes) 000654 1787557072.713236 MIRROR_RESYNC 3 mirrors 000655 1787557073.904588 mapread 0x02e8dd thru 0x030b06 (0x222a bytes) 000656 1787557073.936547 mapwrite 0x026693 thru 0x031e26 (0xb794 bytes) 000658 1787557074.083921 punch from 0x01af6d to 0x020f24 (0x5fb8 bytes) 000660 1787557074.093558 trunc from 0x01f6ec to 0x027224 (0x7b39 bytes) 000662 1787557074.098678 fallocate 0x02e600 thru 0x02f31f (0xd20 bytes) 000663 1787557074.115501 mapwrite 0x02ba61 thru 0x0322a8 (0x6848 bytes) 000664 1787557074.145996 read 0x0098d3 thru 0x010e8c (0x75ba bytes) 000665 1787557074.165532 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x374c5): Mirror 9: 0xb1f28af8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x374c5): Mirror 9: 0xb1f28af8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x374c5: skipped 000666 1787557074.475252 trunc from 0x0374c5 to 0x004c38 (0x3288c bytes) 000667 1787557074.504052 write 0x01b08a thru 0x01ba70 (0x9e7 bytes) 000668 1787557074.507085 write 0x0135ae thru 0x0188bd (0x5310 bytes) 000669 1787557074.550726 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ba71): Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ba71): Mirror 9: 0x100920f0 (stale) CRC-32 checksum value for chunk [0, 0x1ba71): Mirror 1: 0x5b88c6d Mirror 10: 0x5b88c6d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ba71: skipped 000670 1787557075.571366 mapwrite 0x00456b thru 0x00c8d1 (0x8367 bytes) 000671 1787557075.688174 fallocate 0x004850 thru 0x00cbb4 (0x8365 bytes) 000672 1787557075.694087 write 0x00881a thru 0x01775f (0xef46 bytes) 000673 1787557075.711834 mapread 0x00937a thru 0x016233 (0xceba bytes) 000675 1787557075.719286 read 0x011466 thru 0x01ba70 (0xa60b bytes) 000676 1787557075.727488 read 0x0085ce thru 0x013e26 (0xb859 bytes) 000677 1787557075.728479 mapread 0x004aed thru 0x00b8a1 (0x6db5 bytes) 000678 1787557075.731313 trunc from 0x01ba71 to 0x033a30 (0x17fc0 bytes) 000679 1787557075.761525 trunc from 0x033a31 to 0x01cddb (0x16c55 bytes) 000680 1787557075.777094 mapread 0x006395 thru 0x00a3a4 (0x4010 bytes) 000681 1787557075.778535 fallocate 0x01a731 thru 0x024e6e (0xa73e bytes) 000682 1787557075.784816 mapread 0x022d41 thru 0x024e6e (0x212e bytes) 000684 1787557075.792510 fallocate 0x0274cf thru 0x02a638 (0x316a bytes) 000685 1787557075.798047 mapread 0x0146c6 thru 0x01a431 (0x5d6c bytes) 000686 1787557075.801902 mapread 0x0028ef thru 0x005ce5 (0x33f7 bytes) 000687 1787557075.806555 mapwrite 0x032ebb thru 0x03ae96 (0x7fdc bytes) 000688 1787557075.902208 read 0x014e65 thru 0x02418c (0xf328 bytes) 000689 1787557075.902871 mapread 0x032e63 thru 0x03ae96 (0x8034 bytes) 000690 1787557075.907034 trunc from 0x024104 to 0x03246b (0xe368 bytes) 000691 1787557075.931374 MIRROR_RESYNC 3 mirrors 000692 1787557076.773794 trunc from 0x03ae97 to 0x01668b (0x2480b bytes) 000693 1787557076.809874 read 0x005bb9 thru 0x00a7a6 (0x4bee bytes) 000695 1787557076.820368 write 0x036a55 thru 0x03a70d (0x3cb9 bytes) 000696 1787557076.823394 trunc from 0x006c9c to 0x00867d (0x19e2 bytes) 000697 1787557076.833793 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a70e): Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a70e): Mirror 9: 0xf13684e8 (stale) Mirror 10: 0xf13684e8 (stale) CRC-32 checksum value for chunk [0, 0x3a70e): Mirror 1: 0x78f4374 Mirror 11: 0x78f4374 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a70e: skipped 000698 1787557078.042296 write 0x03ca3f thru 0x03d898 (0xe5a bytes) 000699 1787557078.134591 trunc from 0x007540 to 0x0118b1 (0xa372 bytes) 000700 1787557078.148880 punch from 0x011b25 to 0x012879 (0xd55 bytes) 000701 1787557078.167303 punch from 0x031623 to 0x03d898 (0xc276 bytes) 000702 1787557078.200766 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d899): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d899): Mirror 9: 0x82d209d9 (stale) Mirror 10: 0x82d209d9 (stale) Mirror 11: 0x87e53e16 (stale) CRC-32 checksum value for chunk [0, 0x3d899): Mirror 1: 0x148ba51a Mirror 12: 0x148ba51a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d899: skipped 000703 1787557079.643160 trunc from 0x00e28d to 0x019f4d (0xbcc1 bytes) 000705 1787557079.701392 mapread 0x01b898 thru 0x022efc (0x7665 bytes) 000706 1787557079.728311 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d899): Mirror 9: 0x82d209d9 (stale) Mirror 10: 0x82d209d9 (stale) Mirror 12: 0x148ba51a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d899): Mirror 9: 0x82d209d9 (stale) Mirror 10: 0x82d209d9 (stale) Mirror 12: 0x148ba51a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d899: skipped 000707 1787557080.129389 fallocate 0x0048e8 thru 0x0145b2 (0xfccb bytes) 000708 1787557080.137131 trunc from 0x02b77f to 0x039793 (0xe015 bytes) 000709 1787557080.146672 trunc from 0x00d012 to 0x0124d5 (0x54c4 bytes) 000710 1787557080.156072 mapread 0x03357f thru 0x035596 (0x2018 bytes) 000711 1787557080.195454 write 0x027383 thru 0x02aef8 (0x3b76 bytes) 000712 1787557080.210781 trunc from 0x02e2a8 to 0x02fa98 (0x17f1 bytes) 000713 1787557080.216864 mapwrite 0x00e043 thru 0x01a1b4 (0xc172 bytes) 000714 1787557080.260804 punch from 0x01e685 to 0x02d2a8 (0xec24 bytes) 000715 1787557080.270655 trunc from 0x03d899 to 0x01802b (0x2586d bytes) 000716 1787557080.288429 read 0x00e21e thru 0x01802b (0x9e0e bytes) 000717 1787557080.289233 mapread 0x00f486 thru 0x012562 (0x30dd bytes) 000718 1787557080.290247 read 0x010ae3 thru 0x01802b (0x7549 bytes) 000719 1787557080.293269 mapwrite 0x02ccec thru 0x03b894 (0xeba9 bytes) 000720 1787557080.341586 mapwrite 0x00fa64 thru 0x010eb7 (0x1454 bytes) 000721 1787557080.356651 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b895): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b895): Mirror 9: 0x312a3974 (stale) Mirror 10: 0x312a3974 (stale) Mirror 12: 0xfe05dd8c (stale) CRC-32 checksum value for chunk [0, 0x3b895): Mirror 1: 0xa3541f1c Mirror 13: 0xa3541f1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b895: skipped 000723 1787557081.112883 mapread 0x015b5c thru 0x01aa74 (0x4f19 bytes) 000724 1787557081.146756 fallocate 0x024a0a thru 0x029f9e (0x5595 bytes) 000726 1787557081.187373 trunc from 0x0138e4 to 0x01fc85 (0xc3a2 bytes) 000727 1787557081.194552 write 0x03da1b thru 0x03ffff (0x25e5 bytes) 000728 1787557081.201474 trunc from 0x040000 to 0x01847a (0x27b85 bytes) 000729 1787557081.256236 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1847b): Mirror 9: 0x63e81a90 (stale) Mirror 10: 0x63e81a90 (stale) Mirror 12: 0xe70b3b1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1847b): Mirror 9: 0x63e81a90 (stale) Mirror 10: 0x63e81a90 (stale) Mirror 12: 0xe70b3b1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1847b: skipped 000731 1787557081.587952 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1847b): Mirror 10: 0x63e81a90 (stale) Mirror 12: 0xe70b3b1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1847b): Mirror 10: 0x63e81a90 (stale) Mirror 12: 0xe70b3b1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1847b: skipped 000733 1787557081.906009 write 0x021df7 thru 0x026e09 (0x5013 bytes) 000734 1787557081.918353 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26e0a): Mirror 10: 0x7a9d0449 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e0a): Mirror 10: 0x7a9d0449 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e0a: skipped 000735 1787557082.191398 mapwrite 0x010655 thru 0x01a444 (0x9df0 bytes) 000736 1787557082.235027 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26e0a): Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e0a): Mirror 10: 0x7a9d0449 (stale) CRC-32 checksum value for chunk [0, 0x26e0a): Mirror 1: 0xf908e671 Mirror 11: 0xf908e671 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e0a: skipped 000738 1787557083.158241 mapwrite 0x02dd2e thru 0x038a22 (0xacf5 bytes) 000739 1787557083.347585 mapwrite 0x005337 thru 0x00682e (0x14f8 bytes) 000740 1787557083.382284 fallocate 0x0003a0 thru 0x009008 (0x8c69 bytes) 000741 1787557083.389583 write 0x0386ea thru 0x03e924 (0x623b bytes) 000742 1787557083.390778 mapread 0x011d1c thru 0x01e54d (0xc832 bytes) 000743 1787557083.443760 mapread 0x019824 thru 0x0272f1 (0xdace bytes) 000744 1787557083.457235 trunc from 0x03e925 to 0x0076bf (0x37265 bytes) 000745 1787557083.499024 MIRROR_RESYNC 3 mirrors 000746 1787557084.262326 mapwrite 0x01818a thru 0x02185e (0x96d5 bytes) 000748 1787557084.361206 fallocate 0x037d70 thru 0x03d34f (0x55e0 bytes) 000749 1787557084.372236 write 0x005101 thru 0x006632 (0x1532 bytes) 000750 1787557084.409539 mapwrite 0x03ab97 thru 0x03ec77 (0x40e1 bytes) 000752 1787557084.463043 fallocate 0x037d0e thru 0x03f4da (0x77cd bytes) 000753 1787557084.469263 trunc from 0x016086 to 0x0226ab (0xc626 bytes) 000754 1787557084.474854 mapwrite 0x009571 thru 0x00bc0d (0x269d bytes) 000755 1787557084.503933 write 0x00f841 thru 0x018859 (0x9019 bytes) 000756 1787557084.518139 punch from 0x03c4ee to 0x03f4da (0x2fed bytes) 000757 1787557084.526130 trunc from 0x03f4db to 0x02cf3e (0x1259c bytes) 000758 1787557084.540853 read 0x00edbc thru 0x01d751 (0xe996 bytes) 000760 1787557084.549785 punch from 0x020a05 to 0x02cf3e (0xc53a bytes) 000761 1787557084.554543 write 0x00d906 thru 0x0154d0 (0x7bcb bytes) 000762 1787557084.555868 trunc from 0x015ca0 to 0x01e886 (0x8be7 bytes) 000763 1787557084.569595 mapread 0x00f5df thru 0x0105b2 (0xfd4 bytes) 000764 1787557084.570514 mapwrite 0x0136bd thru 0x016036 (0x297a bytes) 000765 1787557084.594107 punch from 0x008b1e to 0x009a88 (0xf6b bytes) 000766 1787557084.601399 trunc from 0x02cf3f to 0x021d18 (0xb226 bytes) 000767 1787557084.621051 mapwrite 0x033617 thru 0x03adb6 (0x77a0 bytes) 000768 1787557084.683954 mapread 0x020802 thru 0x021a1a (0x1219 bytes) 000769 1787557084.692452 mapread 0x02b61c thru 0x0316b4 (0x6099 bytes) 000770 1787557084.701313 mapwrite 0x0246c3 thru 0x030607 (0xbf45 bytes) 000771 1787557084.723719 trunc from 0x021448 to 0x02ee9d (0xda56 bytes) 000772 1787557084.732142 write 0x01c792 thru 0x024f3f (0x87ae bytes) 000773 1787557084.740026 punch from 0x02fee2 to 0x030e0b (0xf2a bytes) 000774 1787557084.751046 write 0x010b28 thru 0x01b71a (0xabf3 bytes) 000775 1787557084.755045 mapwrite 0x02c0d9 thru 0x0308bb (0x47e3 bytes) 000776 1787557084.801173 trunc from 0x03adb7 to 0x02615b (0x14c5b bytes) 000777 1787557084.819309 read 0x01de73 thru 0x0247f6 (0x6984 bytes) 000778 1787557084.820191 mapread 0x00c221 thru 0x012cda (0x6aba bytes) 000779 1787557084.823352 read 0x020781 thru 0x025d43 (0x55c3 bytes) 000781 1787557084.830877 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2615c): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2615c): Mirror 10: 0xb0e6b3d5 (stale) Mirror 11: 0xb0e6b3d5 (stale) CRC-32 checksum value for chunk [0, 0x2615c): Mirror 1: 0x8550ac57 Mirror 12: 0x8550ac57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2615c: skipped 000782 1787557085.872433 mapread 0x01ffe2 thru 0x022cc7 (0x2ce6 bytes) 000783 1787557085.920863 MIRROR_RESYNC 4 mirrors 000784 1787557087.038184 read 0x016c92 thru 0x01b251 (0x45c0 bytes) 000785 1787557087.046382 fallocate 0x039f51 thru 0x03e93d (0x49ed bytes) 000786 1787557087.098283 mapread 0x002191 thru 0x01007e (0xdeee bytes) 000787 1787557087.122205 trunc from 0x01d947 to 0x02615b (0x8815 bytes) 000789 1787557087.130850 trunc from 0x02615c to 0x003422 (0x22d39 bytes) 000790 1787557087.152936 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3423): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3423): Mirror 10: 0x3eef9d1f (stale) Mirror 11: 0x3eef9d1f (stale) Mirror 12: 0x3eef9d1f (stale) CRC-32 checksum value for chunk [0, 0x3423): Mirror 1: 0x3eef9d1f Mirror 13: 0x3eef9d1f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3423: skipped 000791 1787557087.963519 write 0x0053e5 thru 0x011c0d (0xc829 bytes) 000792 1787557088.036580 punch from 0x00f8ce to 0x011c0d (0x2340 bytes) 000793 1787557088.055013 trunc from 0x011c0e to 0x004fe2 (0xcc2b bytes) 000794 1787557088.086335 MIRROR_RESYNC 5 mirrors 000795 1787557089.067030 fallocate 0x00014d thru 0x007ea6 (0x7d5a bytes) 000796 1787557089.093561 fallocate 0x01c829 thru 0x02afa2 (0xe77a bytes) 000797 1787557089.100063 trunc from 0x02afa3 to 0x004aad (0x264f5 bytes) 000798 1787557089.127588 MIRROR_RESYNC 5 mirrors 000799 1787557090.599083 write 0x00eedb thru 0x0145e0 (0x5706 bytes) 000801 1787557090.651271 trunc from 0x0145e1 to 0x00d1f0 (0x73f0 bytes) 000802 1787557090.672041 write 0x01cd84 thru 0x021288 (0x4505 bytes) 000803 1787557090.674689 punch from 0x01a971 to 0x01e2a6 (0x3936 bytes) 000804 1787557090.697899 write 0x0294e7 thru 0x0340c4 (0xabde bytes) 000805 1787557090.702531 trunc from 0x00aae7 to 0x015466 (0xa980 bytes) 000807 1787557090.710427 trunc from 0x0340c5 to 0x009c89 (0x2a43b bytes) 000808 1787557090.733425 punch from 0x0088d8 to 0x009c89 (0x13b2 bytes) 000810 1787557090.739175 punch from 0x000687 to 0x009c89 (0x9603 bytes) 000811 1787557090.745912 trunc from 0x0074aa to 0x009c89 (0x27e0 bytes) 000812 1787557090.753004 mapwrite 0x02465b thru 0x032a77 (0xe41d bytes) 000813 1787557090.867168 fallocate 0x03b428 thru 0x03ffff (0x4bd8 bytes) 000814 1787557090.880333 trunc from 0x00703e to 0x015538 (0xe4fb bytes) 000815 1787557090.892789 read 0x012164 thru 0x017748 (0x55e5 bytes) 000816 1787557090.912352 fallocate 0x00a8af thru 0x018a67 (0xe1b9 bytes) 000817 1787557090.918408 mapwrite 0x0178a3 thru 0x020641 (0x8d9f bytes) 000818 1787557090.987282 punch from 0x00da18 to 0x012d41 (0x532a bytes) 000819 1787557091.000305 read 0x010473 thru 0x01cd08 (0xc896 bytes) 000820 1787557091.039825 mapread 0x01f664 thru 0x023b02 (0x449f bytes) 000821 1787557091.041578 fallocate 0x0008f9 thru 0x0050a6 (0x47ae bytes) 000822 1787557091.058652 mapread 0x024f16 thru 0x02d67b (0x8766 bytes) 000824 1787557091.062187 read 0x029011 thru 0x032a77 (0x9a67 bytes) 000825 1787557091.069455 fallocate 0x01b24c thru 0x024c58 (0x9a0d bytes) 000826 1787557091.088323 read 0x001f9b thru 0x004571 (0x25d7 bytes) 000827 1787557091.095202 mapread 0x019db4 thru 0x0213cc (0x7619 bytes) 000828 1787557091.103068 punch from 0x01feb6 to 0x023dad (0x3ef8 bytes) 000829 1787557091.135647 read 0x0112bd thru 0x01800e (0x6d52 bytes) 000830 1787557091.136326 mapread 0x00a9b6 thru 0x00c894 (0x1edf bytes) 000831 1787557091.149110 MIRROR_RESYNC 5 mirrors 000832 1787557092.749543 MIRROR_RESYNC 5 mirrors 000833 1787557093.516636 punch from 0x02c6dc to 0x032a77 (0x639c bytes) 000834 1787557093.558654 read 0x022aba thru 0x023893 (0xdda bytes) 000835 1787557093.568444 write 0x03853a thru 0x03ffff (0x7ac6 bytes) 000836 1787557093.570282 mapwrite 0x036c73 thru 0x039950 (0x2cde bytes) 000837 1787557093.612170 punch from 0x03c601 to 0x03ffff (0x39ff bytes) 000838 1787557093.638210 punch from 0x0342ad to 0x03b066 (0x6dba bytes) 000839 1787557093.656576 trunc from 0x027009 to 0x0271eb (0x1e3 bytes) 000840 1787557093.672577 write 0x02f82e thru 0x039056 (0x9829 bytes) 000841 1787557093.715683 fallocate 0x02a648 thru 0x033489 (0x8e42 bytes) 000842 1787557093.727989 read 0x022dd8 thru 0x0239c9 (0xbf2 bytes) 000843 1787557093.730858 mapread 0x010a20 thru 0x018996 (0x7f77 bytes) 000844 1787557093.752283 mapwrite 0x03620e thru 0x03ffff (0x9df2 bytes) 000845 1787557093.844784 trunc from 0x02861f to 0x030170 (0x7b52 bytes) 000846 1787557093.857941 write 0x0299da thru 0x02e315 (0x493c bytes) 000847 1787557093.879875 read 0x028710 thru 0x02b8f7 (0x31e8 bytes) 000848 1787557093.889063 mapread 0x02f516 thru 0x036750 (0x723b bytes) 000849 1787557093.890903 mapread 0x01491a thru 0x01f5c8 (0xacaf bytes) 000852 1787557093.892664 fallocate 0x035219 thru 0x03ffff (0xade7 bytes) 000853 1787557093.899168 trunc from 0x040000 to 0x01350a (0x2caf5 bytes) 000854 1787557093.918731 read 0x005943 thru 0x0087cb (0x2e89 bytes) 000855 1787557093.921343 write 0x01b61d thru 0x01c923 (0x1307 bytes) 000856 1787557093.929271 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1c924): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c924): Mirror 10: 0xed94b048 (stale) Mirror 11: 0xed94b048 (stale) Mirror 12: 0xed94b048 (stale) Mirror 13: 0xed94b048 (stale) CRC-32 checksum value for chunk [0, 0x1c924): Mirror 1: 0x2e4d6a88 Mirror 14: 0x2e4d6a88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c924: skipped 000858 1787557095.203879 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1c924): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c924): Mirror 10: 0xed94b048 (stale) Mirror 11: 0xed94b048 (stale) Mirror 12: 0xed94b048 (stale) Mirror 13: 0xed94b048 (stale) CRC-32 checksum value for chunk [0, 0x1c924): Mirror 1: 0x2e4d6a88 Mirror 14: 0x2e4d6a88 Mirror 15: 0x2e4d6a88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c924: skipped 000859 1787557096.253239 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c924): Mirror 10: 0xed94b048 (stale) Mirror 12: 0xed94b048 (stale) Mirror 13: 0xed94b048 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c924): Mirror 10: 0xed94b048 (stale) Mirror 12: 0xed94b048 (stale) Mirror 13: 0xed94b048 (stale) CRC-32 checksum value for chunk [0, 0x1c924): Mirror 1: 0x2e4d6a88 Mirror 14: 0x2e4d6a88 Mirror 15: 0x2e4d6a88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c924: skipped 000860 1787557097.351535 MIRROR_RESYNC 6 mirrors 000862 1787557098.590744 punch from 0x00631f to 0x0089a1 (0x2683 bytes) 000864 1787557098.693535 read 0x01a85a thru 0x01c923 (0x20ca bytes) 000865 1787557098.720811 mapwrite 0x03f0ee thru 0x03fbb4 (0xac7 bytes) 000866 1787557098.862038 punch from 0x0261c0 to 0x0348bf (0xe700 bytes) 000867 1787557098.872750 trunc from 0x03fbb5 to 0x00c4a6 (0x3370e bytes) 000868 1787557098.931386 trunc from 0x00c4a7 to 0x01d188 (0x10ce2 bytes) 000869 1787557098.965570 fallocate 0x02f601 thru 0x03e641 (0xf041 bytes) 000870 1787557098.976565 write 0x007d51 thru 0x015d12 (0xdfc2 bytes) 000871 1787557099.005684 MIRROR_RESYNC 6 mirrors 000872 1787557101.155542 trunc from 0x012ef1 to 0x022651 (0xf761 bytes) 000873 1787557101.225817 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e642): Mirror 10: 0xcd731027 (stale) Mirror 12: 0xcd731027 (stale) Mirror 13: 0xcd731027 (stale) Mirror 14: 0xcd731027 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e642): Mirror 10: 0xcd731027 (stale) Mirror 12: 0xcd731027 (stale) Mirror 13: 0xcd731027 (stale) Mirror 14: 0xcd731027 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e642: skipped 000874 1787557101.583842 mapread 0x005978 thru 0x00fb08 (0xa191 bytes) 000875 1787557101.600757 write 0x013d60 thru 0x021d80 (0xe021 bytes) 000876 1787557101.656484 mapwrite 0x02f07c thru 0x03e60a (0xf58f bytes) 000877 1787557101.786801 write 0x000e2c thru 0x003a7f (0x2c54 bytes) 000878 1787557101.795921 trunc from 0x03e642 to 0x02addc (0x13865 bytes) 000880 1787557101.824742 write 0x01d7fd thru 0x0298e0 (0xc0e4 bytes) 000881 1787557101.832666 punch from 0x01f5e0 to 0x0226a9 (0x30ca bytes) 000882 1787557101.868269 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2addd): Mirror 12: 0xd00d9c27 (stale) Mirror 13: 0xd00d9c27 (stale) Mirror 14: 0xd00d9c27 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2addd): Mirror 12: 0xd00d9c27 (stale) Mirror 13: 0xd00d9c27 (stale) Mirror 14: 0xd00d9c27 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2addd: skipped 000883 1787557102.450899 mapwrite 0x027d31 thru 0x02bd4b (0x401b bytes) 000884 1787557102.594196 fallocate 0x019b6f thru 0x021e2e (0x82c0 bytes) 000885 1787557102.624836 MIRROR_RESYNC 4 mirrors 000886 1787557104.509229 mapwrite 0x03e57a thru 0x03e6d1 (0x158 bytes) 000889 1787557104.706089 mapwrite 0x0222cb thru 0x0268dc (0x4612 bytes) 000890 1787557104.807033 write 0x005d19 thru 0x0156dd (0xf9c5 bytes) 000891 1787557104.856281 fallocate 0x02e0f3 thru 0x03233c (0x424a bytes) 000892 1787557104.896092 mapread 0x03948c thru 0x03e6d1 (0x5246 bytes) 000893 1787557105.010025 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e6d2): Mirror 13: 0xec2c46b2 (stale) Mirror 14: 0xec2c46b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e6d2): Mirror 13: 0xec2c46b2 (stale) Mirror 14: 0xec2c46b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e6d2: skipped 000894 1787557106.352774 punch from 0x033ef4 to 0x03e6d1 (0xa7de bytes) 000895 1787557106.367326 trunc from 0x00f1a0 to 0x01d2b4 (0xe115 bytes) 000896 1787557106.398702 mapwrite 0x0303bb thru 0x03441b (0x4061 bytes) 000897 1787557106.516168 trunc from 0x03e6d2 to 0x0340ed (0xa5e4 bytes) 000898 1787557106.559233 write 0x0188f5 thru 0x01cf37 (0x4643 bytes) 000899 1787557106.581587 write 0x01176d thru 0x01248e (0xd22 bytes) 000900 1787557106.615804 read 0x014bd0 thru 0x01fb9e (0xafcf bytes) 000901 1787557106.649360 fallocate 0x03015a thru 0x03eb61 (0xea08 bytes) 000902 1787557106.685113 fallocate 0x019584 thru 0x028efc (0xf979 bytes) 000903 1787557106.703201 fallocate 0x02591e thru 0x0287aa (0x2e8d bytes) 000905 1787557106.710830 fallocate 0x012a0e thru 0x01383d (0xe30 bytes) 000906 1787557106.725551 mapread 0x02bb91 thru 0x02c425 (0x895 bytes) 000907 1787557106.727976 read 0x01d8f2 thru 0x02c813 (0xef22 bytes) 000908 1787557106.758293 write 0x01af92 thru 0x01b4fd (0x56c bytes) 000909 1787557106.766126 fallocate 0x00ecdd thru 0x00fa06 (0xd2a bytes) 000910 1787557106.780033 trunc from 0x01c4e7 to 0x01c95d (0x477 bytes) 000911 1787557106.816383 punch from 0x023a52 to 0x02b5c8 (0x7b77 bytes) 000912 1787557106.842798 MIRROR_RESYNC 3 mirrors 000913 1787557109.111227 mapwrite 0x030564 thru 0x03e2cd (0xdd6a bytes) 000914 1787557109.269094 write 0x00441a thru 0x011f9c (0xdb83 bytes) 000915 1787557109.279976 mapread 0x01a81b thru 0x0229b1 (0x8197 bytes) 000917 1787557109.297024 write 0x01125a thru 0x01a801 (0x95a8 bytes) 000918 1787557109.304222 MIRROR_RESYNC 3 mirrors 000919 1787557110.727126 read 0x031951 thru 0x03bc6e (0xa31e bytes) 000920 1787557110.755148 MIRROR_EXTEND to 4 mirrors 000921 1787557111.927488 trunc from 0x0264c8 to 0x032683 (0xc1bc bytes) 000922 1787557111.970050 punch from 0x01fa93 to 0x02d1af (0xd71d bytes) 000923 1787557111.976525 punch from 0x01951e to 0x027a2b (0xe50e bytes) 000924 1787557111.985107 trunc from 0x03eb62 to 0x02b8b6 (0x132ab bytes) 000925 1787557112.018130 trunc from 0x02b8b7 to 0x020ae6 (0xadd0 bytes) 000926 1787557112.046427 write 0x00962b thru 0x018926 (0xf2fc bytes) 000928 1787557112.084039 mapread 0x00831f thru 0x00ef34 (0x6c16 bytes) 000929 1787557112.107478 mapread 0x01a0e4 thru 0x020ae6 (0x6a03 bytes) 000930 1787557112.156580 write 0x008f2d thru 0x00c033 (0x3107 bytes) 000931 1787557112.164703 mapread 0x000c33 thru 0x010915 (0xfce3 bytes) 000932 1787557112.181402 trunc from 0x020ae7 to 0x021729 (0xc43 bytes) 000933 1787557112.245016 write 0x0039bb thru 0x00af73 (0x75b9 bytes) 000934 1787557112.249658 punch from 0x01bc0b to 0x021729 (0x5b1f bytes) 000935 1787557112.268241 MIRROR_RESYNC 4 mirrors 000936 1787557113.646323 trunc from 0x011aa3 to 0x01c91b (0xae79 bytes) 000937 1787557113.717739 mapwrite 0x0311fb thru 0x03bab7 (0xa8bd bytes) 000938 1787557113.896168 punch from 0x01f0ba to 0x0273e9 (0x8330 bytes) 000939 1787557113.902318 mapread 0x02db2f thru 0x02dfd0 (0x4a2 bytes) 000940 1787557113.903190 trunc from 0x024049 to 0x02e073 (0xa02b bytes) 000942 1787557113.915100 fallocate 0x029af1 thru 0x02fb08 (0x6018 bytes) 000943 1787557113.921137 mapread 0x028f55 thru 0x02d9cc (0x4a78 bytes) 000944 1787557113.989402 read 0x037993 thru 0x03bab7 (0x4125 bytes) 000945 1787557113.990037 punch from 0x03589a to 0x03bab7 (0x621e bytes) 000946 1787557113.999622 punch from 0x0035a0 to 0x00d33a (0x9d9b bytes) 000947 1787557114.007128 trunc from 0x03bab8 to 0x02c43c (0xf67b bytes) 000948 1787557114.053883 read 0x014e14 thru 0x015a60 (0xc4d bytes) 000949 1787557114.069787 trunc from 0x020af4 to 0x02ae78 (0xa385 bytes) 000950 1787557114.076968 read 0x01a59b thru 0x022dfb (0x8861 bytes) 000951 1787557114.088750 punch from 0x0128a1 to 0x016869 (0x3fc9 bytes) 000954 1787557114.099729 punch from 0x0192c3 to 0x0243bc (0xb0fa bytes) 000955 1787557114.126382 mapwrite 0x0113c8 thru 0x0166e0 (0x5319 bytes) 000956 1787557114.231777 fallocate 0x0227c0 thru 0x030194 (0xd9d5 bytes) 000957 1787557114.244621 write 0x024923 thru 0x026676 (0x1d54 bytes) 000959 1787557114.261906 write 0x0251ea thru 0x034e78 (0xfc8f bytes) 000960 1787557114.270521 trunc from 0x034e79 to 0x022de6 (0x12092 bytes) 000961 1787557114.325987 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22de7): Mirror 13: 0xaaa3b01a (stale) Mirror 14: 0xaaa3b01a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22de7): Mirror 13: 0xaaa3b01a (stale) Mirror 14: 0xaaa3b01a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22de7: skipped 000962 1787557115.058192 mapwrite 0x004221 thru 0x00cdfd (0x8bdd bytes) 000963 1787557115.139971 trunc from 0x022de7 to 0x01affe (0x7de8 bytes) 000964 1787557115.235049 write 0x02fd4e thru 0x03dcf8 (0xdfab bytes) 000965 1787557115.245295 MIRROR_RESYNC 3 mirrors 000966 1787557116.988465 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b6:0x0]: Invalid argument (22) 000967 1787557117.403019 trunc from 0x03dcf9 to 0x00c4e6 (0x31812 bytes) 000968 1787557117.492582 punch from 0x009299 to 0x00c4e6 (0x324e bytes) 000969 1787557117.498059 mapwrite 0x005e68 thru 0x006cf4 (0xe8d bytes) 000971 1787557117.518034 write 0x01b4bb thru 0x0276b8 (0xc1fe bytes) 000972 1787557117.520746 mapwrite 0x02f84f thru 0x039244 (0x99f6 bytes) 000973 1787557117.570494 trunc from 0x039245 to 0x02d259 (0xbfeb bytes) 000974 1787557117.586524 trunc from 0x02d25a to 0x0396f3 (0xc49a bytes) 000975 1787557117.606093 write 0x0008e5 thru 0x00ec09 (0xe325 bytes) 000976 1787557117.620757 MIRROR_RESYNC 2 mirrors 000977 1787557118.535030 MIRROR_RESYNC 2 mirrors 000978 1787557119.187835 mapwrite 0x0243b3 thru 0x028549 (0x4197 bytes) 000979 1787557119.359321 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x396f4): Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396f4): Mirror 13: 0x392726f0 (stale) CRC-32 checksum value for chunk [0, 0x396f4): Mirror 1: 0x81e4e95f Mirror 14: 0x81e4e95f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396f4: skipped 000980 1787557120.876196 trunc from 0x0396f4 to 0x02edbc (0xa937 bytes) 000981 1787557120.979938 trunc from 0x02edbd to 0x023eb9 (0xaf03 bytes) 000982 1787557121.016288 punch from 0x0018d5 to 0x005212 (0x393e bytes) 000983 1787557121.030727 punch from 0x02258a to 0x023eb9 (0x1930 bytes) 000985 1787557121.057739 fallocate 0x01b86e thru 0x01be3b (0x5ce bytes) 000986 1787557121.066848 write 0x030a3b thru 0x03ef07 (0xe4cd bytes) 000987 1787557121.077722 punch from 0x005e91 to 0x007ba5 (0x1d15 bytes) 000988 1787557121.092096 mapread 0x02d884 thru 0x03a9d3 (0xd150 bytes) 000989 1787557121.139937 read 0x01ad81 thru 0x0233c6 (0x8646 bytes) 000990 1787557121.171484 read 0x0212af thru 0x02990c (0x865e bytes) 000991 1787557121.199263 punch from 0x01bc59 to 0x0286d2 (0xca7a bytes) 000992 1787557121.205819 mapwrite 0x02483d thru 0x031f71 (0xd735 bytes) 000993 1787557121.326458 trunc from 0x03ef08 to 0x011d47 (0x2d1c0 bytes) 000994 1787557121.379916 trunc from 0x011d48 to 0x03d167 (0x2b420 bytes) 000995 1787557121.415368 punch from 0x013482 to 0x01af23 (0x7aa2 bytes) 000996 1787557121.436132 fallocate 0x02717e thru 0x02cb00 (0x5983 bytes) 000997 1787557121.444930 write 0x03e145 thru 0x03ffff (0x1ebb bytes) 000998 1787557121.447113 trunc from 0x009603 to 0x019397 (0xfd95 bytes) 000999 1787557121.464028 mapwrite 0x02a161 thru 0x030b42 (0x69e2 bytes) 001000 1787557121.559479 write 0x024ad6 thru 0x02bfea (0x7515 bytes) All operations completed A-OK! PASS 70a (117s) == sanity-flr test 71: check mirror extend parent fid ==== 03:38:48 (1787557128) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 03:39:00 (1787557140) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x311:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.252246 s, 8.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x311:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778919 s, 13.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x312:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x311:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x312:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x311:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 03:39:10 (1787557150) Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre2 Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3670176 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7932800 .. Extending file size to 8948672 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9142560 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9041024 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61371 61372 61374 61375 61378 done 192.168.206.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg606-client.virtnet /mnt/lustre2 (opts:) 192.168.206.106@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg606-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:40:33 (1787557233) Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre2 Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4783008 .. done Extending file size to 2558976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7209856 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7078816 .. Extending file size to 9078432 .. Extending file size to 7439392 .. Extending file size to 8926752 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62730 62731 62734 62735 62736 done 192.168.206.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg606-client.virtnet /mnt/lustre2 (opts:) 192.168.206.106@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg606-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:41:57 (1787557317) Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2572 1268732 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17708 3589312 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25404 3581616 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43112 7170928 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2572 1268732 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17708 3589312 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25404 3581616 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43112 7170928 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148362 s, 14.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216058 s, 9.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xe5fffe40 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg606-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:42:19 (1787557339) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137056 s, 15.3 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 03:42:27 (1787557347) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x206:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x328:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x328:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x329:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x308:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:42:35 (1787557355) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132635 s, 15.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:42:44 (1787557364) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131035 s, 16.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x32b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x309:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x32a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x32b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x309:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x30a:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:42:53 (1787557373) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124328 s, 16.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x32c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x30b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x30c:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:43:26 (1787557406) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg606-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143682 s, 7.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:44:20 (1787557460) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg606-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x32:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x33:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199513 s, 10.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x32:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x33:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x32:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x33:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:45:13 (1787557513) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg606-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg606-server: Pool lustre.test_204f created Adding targets to pool oleg606-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg606-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0996176 s, 21.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x34:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x31:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x34:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x31:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg606-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg606-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg606-server: Pool lustre.test_204f destroyed PASS 204f (70s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:46:22 (1787557582) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:46:30 (1787557590) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:46:39 (1787557599) oleg606-server: Pool lustre.test_206 created Adding targets to pool oleg606-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg606-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg606-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg606-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg606-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:47:10 (1787557630) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836531 s, 12.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673877 s, 15.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf8ca62c0 Mirror 3: 0xf8ca62c0 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:47:22 (1787557642) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (6s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:47:27 (1787557647) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:47:32 (1787557652) Starting client: oleg606-client.virtnet: -o user_xattr,flock 192.168.206.106@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0690903 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2200000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2290000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2290000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2290000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1b90000 Read this (4 bytes): BBBB 192.168.206.106@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg606-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:48:15 (1787557695) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.032561 s, 252 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:48:24 (1787557704) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11064 s, 9.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x223 lmm_fid: [0x200000405:0x223:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x223 lmm_fid: [0x200000405:0x223:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x336:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611683 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 03:48:26.000000000 -0400 Modify: 2026-08-24 03:48:26.000000000 -0400 Change: 2026-08-24 03:48:28.000000000 -0400 Birth: 2026-08-24 03:48:25.000000000 -0400 PASS 210a (12s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:48:36 (1787557716) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492547 s, 21.3 MB/s oleg606-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg606-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg606-server: O/280000401/d23/337: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:48:56 (1787557736) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.293271 s, 140 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0327604 s, 125 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 03:48 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x228:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 03:48 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:49:05 (1787557745) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:49:14 (1787557754) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 1933 sec =========== 03:49:26 (1787557766) === sanity-flr: start cleanup 03:49:27 (1787557767) === === sanity-flr: finish cleanup 03:49:36 (1787557776) ===