-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 08:27:54 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg459-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 08:28:13 (1787574493) === oleg459-client.virtnet: executing check_config_client /mnt/lustre oleg459-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg459-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98a80e47e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff98a80e47e800.idle_timeout=debug disable quota as required oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 08:28:53 (1787574533) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8241] == sanity-flr test 0a: lfs mirror create with -N option == 08:28:55 (1787574535) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:29:11 (1787574551) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:29:14 (1787574554) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:29:18 (1787574558) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:29:44 (1787574584) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:29:48 (1787574588) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 08:29:52 (1787574592) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff98a80e47e800.stats=clear osc.lustre-OST0001-osc-ffff98a80e47e800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:30:02 (1787574602) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:30:15 (1787574615) PASS 0j (13s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:30:27 (1787574627) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:30:31 (1787574631) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:30:39 (1787574639) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 08:30:49 (1787574649) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:30:56 (1787574656) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:31:04 (1787574664) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:31:12 (1787574672) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:31:23 (1787574683) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:31:30 (1787574690) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.10528 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614184 s, 17.1 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:31:39 (1787574699) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.414911 s, 37.9 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:31:46 (1787574706) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:31:56 (1787574716) Stopping /mnt/lustre-ost1 (opts:) on oleg459-server oleg459-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98a80e47e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98a80e47e800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg459-server: oleg459-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg459-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg459-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg459-server oleg459-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98a80e47e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98a80e47e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg459-server: oleg459-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg459-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (132s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:34:08 (1787574848) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.464915 s, 15.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg459-server oleg459-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98a80e47e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98a80e47e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg459-server: oleg459-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg459-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:34:53 (1787574893) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg459-server oleg459-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98a80e47e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98a80e47e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg459-server: oleg459-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg459-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg459-server oleg459-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98a80e47e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98a80e47e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg459-server: oleg459-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg459-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (70s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:36:04 (1787574964) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.68617 s, 19.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 08:36 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.746s user 0m0.000s sys 0m0.155s Stopping /mnt/lustre-ost1 (opts:) on oleg459-server oleg459-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff98a80e47e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98a80e47e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.999s user 0m0.000s sys 0m0.138s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg459-server: oleg459-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg459-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg459-server oleg459-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff98a80e47e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98a80e47e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.957s user 0m0.001s sys 0m0.180s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg459-server: oleg459-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg459-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg459-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98a80e47e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98a80e47e800.ost_server_uuid in FULL state after 0 sec PASS 33b (81s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:37:24 (1787575044) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:37:28 (1787575048) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:37:31 (1787575051) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 08:37:34 (1787575054) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 08:37:42 (1787575062) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.464756 s, 27.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:37:53 (1787575073) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333787 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025858 s, 40.6 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:38:00 (1787575080) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:38:07 (1787575087) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00254191 s, 403 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00201546 s, 508 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000955285 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115074 s, 890 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 08:38:15 (1787575095) create test files with size 15738707 .. 42dcd26333253c9cbd47b2d04ac0af6b - c1d50de68e3d94c92e8f40c0de8490e9 - 97d82ab25734814686d6749a5776e431 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg459-client.virtnet /mnt/lustre (opts:) Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre PASS 37 (18s) == sanity-flr test 38: resync ============================ 08:38:33 (1787575113) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:38:46 (1787575126) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:38:52 (1787575132) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992397 s, 10.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702047 s, 14.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 08:39:01 (1787575141) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.09851 s, 21.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.338902 s, 12.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186698 s, 11.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 08:39:14 (1787575154) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 08:39:18 (1787575158) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:39:21 (1787575161) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376106 s, 27.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 08:39:29 (1787575169) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.275112 s, 11.4 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:39:37 (1787575177) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xab:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:39:46 (1787575186) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.350366 s, 29.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg459-client.virtnet /mnt/lustre (opts:) Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:39:57 (1787575197) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff98a80af9a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.268633 s, 39.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg459-client.virtnet /mnt/lustre (opts:) Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff98a80e47e000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:40:14 (1787575214) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:40:49 (1787575249) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:41:29 (1787575289) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 08:41:34 (1787575294) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.321611 s, 9.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.253625 s, 12.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x796eb2b7 (nosync,stale) Mirror 2: 0x796eb2b7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8f063f7138c4bbf2667ea675a1a18ef8 - mirror 1 checksum: 67d6d5884b8efb1f35c3a3cc35d34048 - mirror 2 checksum: 8f063f7138c4bbf2667ea675a1a18ef8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T08:41:37-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T08:41:37-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8f063f7138c4bbf2667ea675a1a18ef8 - mirror 1 checksum: 67d6d5884b8efb1f35c3a3cc35d34048 - mirror 2 checksum: 67d6d5884b8efb1f35c3a3cc35d34048 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T08:41:37-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T08:41:37-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb6:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:41:46 (1787575306) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281696 s, 37.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:41:55 (1787575315) Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 08:42:07 (1787575327) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 08:42:40 (1787575360) targets are mounted 08:42:40 (1787575360) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269315 s, 38.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb8:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:42:59 (1787575379) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.041318 s, 496 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0372022 s, 551 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:43:10 (1787575390) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.107264 s, 191 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0140605 s, 1.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:43:21 (1787575401) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00605811 s, 2.7 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:43:29 (1787575409) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0745266 s, 275 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:43:38 (1787575418) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00738094 s, 555 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0370262 s, 111 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:43:46 (1787575426) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:44:09 (1787575449) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:44:38 (1787575478) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:45:02 (1787575502) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:45:26 (1787575526) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 08:45:35 (1787575535) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57582 mirror split pid 57583 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57582 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57583 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 08:46:45 (1787575605) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 5607 000001 1787575607.764016 write 0x01e8bc thru 0x0229e9 (0x412e bytes) fallocating to largest ever: 0x169e6 000002 1787575607.786092 fallocate 0x00b020 thru 0x0169e5 (0xb9c6 bytes) 000003 1787575607.799363 mapwrite 0x000c82 thru 0x001b6f (0xeee bytes) truncating to largest ever: 0x1cdb1 000004 1787575607.831045 trunc from 0x0229ea to 0x01cdb0 (0x5c39 bytes) 000005 1787575607.850526 mapread 0x004282 thru 0x00616d (0x1eec bytes) 000007 1787575607.877653 mapwrite 0x02acc2 thru 0x03a8da (0xfc19 bytes) 000009 1787575607.992618 mapread 0x02c095 thru 0x035bac (0x9b18 bytes) fallocating to largest ever: 0x34ea8 000010 1787575608.001850 fallocate 0x02ba2c thru 0x034ea7 (0x947c bytes) 000011 1787575608.013866 trunc from 0x03a8db to 0x0289d4 (0x11f06 bytes) 000012 1787575608.048872 mapwrite 0x009c40 thru 0x00ae54 (0x1215 bytes) 000013 1787575608.100119 mapwrite 0x03afad thru 0x03ffff (0x5053 bytes) fallocating to largest ever: 0x40000 000014 1787575608.208404 fallocate 0x03c9e7 thru 0x03ffff (0x3619 bytes) 000015 1787575608.215368 punch from 0x015c5b to 0x01e53f (0x88e5 bytes) 000016 1787575608.226449 trunc from 0x02a43d to 0x039d86 (0xf94a bytes) 000017 1787575608.241443 mapwrite 0x001cd8 thru 0x008fc4 (0x72ed bytes) 000018 1787575608.303565 mapread 0x023334 thru 0x0299f7 (0x66c4 bytes) 000019 1787575608.328012 MIRROR_EXTEND to 2 mirrors 000020 1787575609.281219 MIRROR_EXTEND to 3 mirrors 000021 1787575610.607951 write 0x007f92 thru 0x00f15b (0x71ca bytes) 000022 1787575610.677228 mapwrite 0x03ffa6 thru 0x03ffff (0x5a bytes) 000024 1787575610.740459 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xeaf5fd50 (stale) Mirror 3: 0xeaf5fd50 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3a773e4a Mirror 4: 0x3a773e4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000025 1787575612.239724 MIRROR_RESYNC 4 mirrors 000026 1787575613.279986 punch from 0x01a448 to 0x01e550 (0x4109 bytes) 000027 1787575613.385246 punch from 0x01ff76 to 0x027afe (0x7b89 bytes) 000028 1787575613.390953 fallocate 0x015b34 thru 0x01f8d8 (0x9da5 bytes) 000029 1787575613.424307 MIRROR_RESYNC 4 mirrors 000030 1787575614.680631 punch from 0x004133 to 0x00cd56 (0x8c24 bytes) 000031 1787575614.720059 write 0x009977 thru 0x00adea (0x1474 bytes) 000032 1787575614.731091 fallocate 0x0395f0 thru 0x03ffff (0x6a10 bytes) 000033 1787575614.736205 fallocate 0x0130c4 thru 0x018354 (0x5291 bytes) 000034 1787575614.741095 trunc from 0x040000 to 0x01ef07 (0x210f8 bytes) 000035 1787575614.757673 write 0x0309f7 thru 0x03bfa0 (0xb5aa bytes) 000036 1787575614.759354 trunc from 0x01e8c1 to 0x0219e2 (0x3122 bytes) 000037 1787575614.764824 fallocate 0x002097 thru 0x009dc0 (0x7d2a bytes) 000038 1787575614.770205 mapread 0x02920a thru 0x036ff9 (0xddf0 bytes) 000039 1787575614.785175 punch from 0x018a41 to 0x024e32 (0xc3f2 bytes) 000040 1787575614.789099 write 0x036f73 thru 0x03b939 (0x49c7 bytes) 000041 1787575614.791573 fallocate 0x002f79 thru 0x01116d (0xe1f5 bytes) 000042 1787575614.796260 trunc from 0x03bfa1 to 0x03dde3 (0x1e43 bytes) 000043 1787575614.813225 punch from 0x00a3ca to 0x00a706 (0x33d bytes) 000044 1787575614.831246 write 0x01540b thru 0x01db4d (0x8743 bytes) 000045 1787575614.843045 write 0x00254b thru 0x00aadf (0x8595 bytes) 000046 1787575614.854600 write 0x029c1a thru 0x0302c9 (0x66b0 bytes) 000047 1787575614.856300 fallocate 0x0066d2 thru 0x010fb5 (0xa8e4 bytes) 000048 1787575614.867150 MIRROR_RESYNC 4 mirrors 000050 1787575616.186384 trunc from 0x03ce31 to 0x03ddc2 (0xf92 bytes) 000051 1787575616.227760 read 0x018516 thru 0x018fdc (0xac7 bytes) 000055 1787575616.241696 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3dde4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dde4): Mirror 2: 0x1c26bbe4 (stale) Mirror 3: 0x1c26bbe4 (stale) Mirror 4: 0x1c26bbe4 (stale) CRC-32 checksum value for chunk [0, 0x3dde4): Mirror 1: 0x1c26bbe4 Mirror 5: 0x1c26bbe4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dde4: skipped 000056 1787575617.603422 read 0x02122f thru 0x0235a6 (0x2378 bytes) 000057 1787575617.638745 punch from 0x004038 to 0x0123f5 (0xe3be bytes) 000059 1787575617.778515 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3dde4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dde4): Mirror 2: 0x1c26bbe4 (stale) Mirror 3: 0x1c26bbe4 (stale) Mirror 4: 0x1c26bbe4 (stale) Mirror 5: 0x1c26bbe4 (stale) CRC-32 checksum value for chunk [0, 0x3dde4): Mirror 1: 0x9f1a8eba Mirror 6: 0x9f1a8eba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dde4: skipped 000060 1787575619.132722 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dde4): Mirror 2: 0x1c26bbe4 (stale) Mirror 3: 0x1c26bbe4 (stale) Mirror 4: 0x1c26bbe4 (stale) Mirror 5: 0x1c26bbe4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dde4): Mirror 2: 0x1c26bbe4 (stale) Mirror 3: 0x1c26bbe4 (stale) Mirror 4: 0x1c26bbe4 (stale) Mirror 5: 0x1c26bbe4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dde4: skipped 000061 1787575619.668040 mapread 0x029c86 thru 0x02f538 (0x58b3 bytes) 000062 1787575619.692745 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dde4): Mirror 3: 0x1c26bbe4 (stale) Mirror 4: 0x1c26bbe4 (stale) Mirror 5: 0x1c26bbe4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dde4): Mirror 3: 0x1c26bbe4 (stale) Mirror 4: 0x1c26bbe4 (stale) Mirror 5: 0x1c26bbe4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dde4: skipped 000063 1787575620.086360 trunc from 0x028b97 to 0x02974e (0xbb8 bytes) 000064 1787575620.106879 trunc from 0x03dde4 to 0x01a461 (0x23982 bytes) 000065 1787575620.127644 mapwrite 0x0037fd thru 0x008db3 (0x55b7 bytes) 000066 1787575620.175462 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a462): Mirror 3: 0xaa103fba (stale) Mirror 4: 0xaa103fba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a462): Mirror 3: 0xaa103fba (stale) Mirror 4: 0xaa103fba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a462: skipped 000067 1787575620.540022 read 0x0142c1 thru 0x01a461 (0x61a1 bytes) 000068 1787575620.551490 mapread 0x01a25f thru 0x01a461 (0x203 bytes) 000069 1787575620.552530 trunc from 0x01a462 to 0x03daa8 (0x23647 bytes) 000071 1787575620.582399 write 0x037ebc thru 0x03ffff (0x8144 bytes) 000072 1787575620.588145 mapread 0x0356b3 thru 0x03ffff (0xa94d bytes) 000073 1787575620.603173 write 0x00f1ed thru 0x015a9e (0x68b2 bytes) 000074 1787575620.611480 read 0x03fcdc thru 0x03ffff (0x324 bytes) 000075 1787575620.612443 read 0x02eef5 thru 0x03d673 (0xe77f bytes) 000076 1787575620.627631 mapread 0x003b04 thru 0x01011f (0xc61c bytes) 000077 1787575620.634405 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7e6dbfd3 (stale) Mirror 4: 0x7e6dbfd3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x58d04cad Mirror 5: 0x58d04cad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000078 1787575621.977381 read 0x019507 thru 0x023053 (0x9b4d bytes) 000079 1787575621.989926 write 0x01c1a7 thru 0x02147b (0x52d5 bytes) 000080 1787575622.107302 read 0x03b2f1 thru 0x03ffff (0x4d0f bytes) 000083 1787575622.130094 read 0x02a3c5 thru 0x02e15a (0x3d96 bytes) 000084 1787575622.131322 punch from 0x0365b8 to 0x03f7e0 (0x9229 bytes) 000085 1787575622.171097 mapread 0x02ae42 thru 0x02cae3 (0x1ca2 bytes) 000086 1787575622.174938 trunc from 0x040000 to 0x0057d8 (0x3a827 bytes) 000087 1787575622.229203 trunc from 0x0057d9 to 0x023d52 (0x1e57a bytes) 000088 1787575622.274489 trunc from 0x023d53 to 0x01ceca (0x6e88 bytes) 000089 1787575622.307608 MIRROR_RESYNC 4 mirrors 000090 1787575623.049570 punch from 0x003d04 to 0x00bd88 (0x8085 bytes) 000091 1787575623.111247 write 0x00ac18 thru 0x017127 (0xc510 bytes) 000092 1787575623.128499 mapwrite 0x03867a thru 0x03cce5 (0x466c bytes) 000093 1787575623.226522 trunc from 0x00187c to 0x0090e1 (0x7866 bytes) 000095 1787575623.234245 fallocate 0x01aeab thru 0x024743 (0x9899 bytes) 000096 1787575623.240402 mapwrite 0x0281a9 thru 0x02c5d1 (0x4429 bytes) 000097 1787575623.279626 mapread 0x03a8b7 thru 0x03cce5 (0x242f bytes) 000098 1787575623.285656 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3cce6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cce6): Mirror 3: 0xe82fb3b6 (stale) Mirror 4: 0xe82fb3b6 (stale) Mirror 5: 0xe82fb3b6 (stale) CRC-32 checksum value for chunk [0, 0x3cce6): Mirror 1: 0x3bf38343 Mirror 6: 0x3bf38343 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cce6: skipped 000099 1787575624.320569 fallocate 0x00cd54 thru 0x0138a7 (0x6b54 bytes) 000100 1787575624.363331 mapwrite 0x006697 thru 0x0120a1 (0xba0b bytes) 000101 1787575624.441240 trunc from 0x03cce6 to 0x01777c (0x25569 bytes) 000102 1787575624.499616 MIRROR_RESYNC 5 mirrors 000103 1787575625.964580 punch from 0x0046df to 0x00abe6 (0x6508 bytes) 000104 1787575626.048644 punch from 0x006648 to 0x00b9f5 (0x53ae bytes) 000106 1787575626.070689 write 0x002c92 thru 0x010516 (0xd885 bytes) 000107 1787575626.104697 fallocate 0x02f6fc thru 0x031f5b (0x2860 bytes) 000108 1787575626.116999 punch from 0x010ec8 to 0x01777c (0x68b5 bytes) 000109 1787575626.158757 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1777d): Mirror 3: 0xebb50097 (stale) Mirror 4: 0xebb50097 (stale) Mirror 5: 0xebb50097 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1777d): Mirror 3: 0xebb50097 (stale) Mirror 4: 0xebb50097 (stale) Mirror 5: 0xebb50097 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1777d: skipped 000110 1787575626.513350 trunc from 0x010c15 to 0x016bb1 (0x5f9d bytes) 000111 1787575626.522410 write 0x03879b thru 0x03d983 (0x51e9 bytes) 000112 1787575626.528586 fallocate 0x02b63f thru 0x02c052 (0xa14 bytes) 000113 1787575626.535473 punch from 0x034da0 to 0x03d983 (0x8be4 bytes) 000114 1787575626.558587 read 0x01b9bf thru 0x028caf (0xd2f1 bytes) 000115 1787575626.571636 trunc from 0x03d984 to 0x03177a (0xc209 bytes) 000116 1787575626.589413 punch from 0x027f48 to 0x028780 (0x839 bytes) 000117 1787575626.596532 punch from 0x0064e2 to 0x00c276 (0x5d95 bytes) 000118 1787575626.623636 trunc from 0x03177b to 0x015217 (0x1c563 bytes) 000119 1787575626.651959 read 0x005326 thru 0x00848c (0x3167 bytes) 000120 1787575626.664819 mapwrite 0x0020c7 thru 0x00d89e (0xb7d8 bytes) 000121 1787575626.691633 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15218): Mirror 3: 0xa3a733a7 (stale) Mirror 4: 0xa3a733a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15218): Mirror 3: 0xa3a733a7 (stale) Mirror 4: 0xa3a733a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15218: skipped 000122 1787575626.995847 punch from 0x0108cb to 0x0133d0 (0x2b06 bytes) 000123 1787575627.001438 read 0x006fae thru 0x008035 (0x1088 bytes) 000124 1787575627.007284 read 0x00a6c0 thru 0x015217 (0xab58 bytes) 000126 1787575627.028884 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15218): Mirror 4: 0xa3a733a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15218): Mirror 4: 0xa3a733a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15218: skipped 000127 1787575627.365451 write 0x015500 thru 0x0193d0 (0x3ed1 bytes) 000129 1787575627.379910 punch from 0x00f5e1 to 0x013272 (0x3c92 bytes) 000130 1787575627.387415 read 0x008975 thru 0x0111ee (0x887a bytes) 000131 1787575627.397323 read 0x0182ce thru 0x0193d0 (0x1103 bytes) 000132 1787575627.398268 trunc from 0x005361 to 0x0112e1 (0xbf81 bytes) 000133 1787575627.403739 mapwrite 0x005885 thru 0x0078cd (0x2049 bytes) 000134 1787575627.432626 mapread 0x000a52 thru 0x0107e8 (0xfd97 bytes) 000135 1787575627.434001 mapwrite 0x024563 thru 0x02542c (0xeca bytes) 000136 1787575627.494415 mapwrite 0x017136 thru 0x024523 (0xd3ee bytes) 000137 1787575627.549400 fallocate 0x0380de thru 0x03dc99 (0x5bbc bytes) 000138 1787575627.561766 mapread 0x01ccaf thru 0x02c8e8 (0xfc3a bytes) 000139 1787575627.577953 mapread 0x032ebb thru 0x03d255 (0xa39b bytes) 000140 1787575627.581457 write 0x007fea thru 0x00cf54 (0x4f6b bytes) 000142 1787575627.589737 trunc from 0x03dc9a to 0x037c55 (0x6044 bytes) 000144 1787575627.629649 write 0x011957 thru 0x021567 (0xfc11 bytes) 000145 1787575627.638272 trunc from 0x037c56 to 0x03a401 (0x27ac bytes) 000146 1787575627.657507 fallocate 0x036e12 thru 0x038177 (0x1366 bytes) 000149 1787575627.678500 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a402): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a402): Mirror 4: 0x8492cadc (stale) CRC-32 checksum value for chunk [0, 0x3a402): Mirror 1: 0xbe0a6076 Mirror 5: 0xbe0a6076 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a402: skipped 000150 1787575628.634119 mapwrite 0x01bffc thru 0x01de60 (0x1e65 bytes) 000151 1787575628.764413 trunc from 0x0254ac to 0x02f37b (0x9ed0 bytes) 000152 1787575628.781945 write 0x02f65c thru 0x03a146 (0xaaeb bytes) 000153 1787575628.810739 MIRROR_RESYNC 3 mirrors 000154 1787575629.844909 write 0x00ba73 thru 0x0100d0 (0x465e bytes) 000155 1787575629.882465 mapread 0x021d20 thru 0x025db4 (0x4095 bytes) 000156 1787575629.907873 punch from 0x035750 to 0x03a401 (0x4cb2 bytes) 000157 1787575629.916404 MIRROR_RESYNC 3 mirrors 000158 1787575630.750621 mapwrite 0x00e373 thru 0x019720 (0xb3ae bytes) 000159 1787575630.911550 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a402): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a402): Mirror 4: 0xe63bf9b9 (stale) Mirror 5: 0xe63bf9b9 (stale) CRC-32 checksum value for chunk [0, 0x3a402): Mirror 1: 0x5dd7de49 Mirror 6: 0x5dd7de49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a402: skipped 000160 1787575631.973078 mapread 0x0175e9 thru 0x02534a (0xdd62 bytes) 000161 1787575632.002666 write 0x01a8b5 thru 0x01d8b6 (0x3002 bytes) 000162 1787575632.043886 read 0x02d52a thru 0x0354ca (0x7fa1 bytes) 000163 1787575632.051359 mapread 0x028d88 thru 0x02a9b8 (0x1c31 bytes) 000164 1787575632.068233 write 0x035c69 thru 0x03ffff (0xa397 bytes) 000165 1787575632.069445 read 0x03e7ca thru 0x03e9c2 (0x1f9 bytes) 000166 1787575632.070012 trunc from 0x039bab to 0x03f1c7 (0x561d bytes) 000167 1787575632.106245 MIRROR_RESYNC 4 mirrors 000168 1787575633.379164 trunc from 0x040000 to 0x0213c7 (0x1ec38 bytes) 000169 1787575633.442841 read 0x01b5dd thru 0x0213c7 (0x5deb bytes) 000170 1787575633.452242 write 0x009a5e thru 0x009c46 (0x1e9 bytes) 000172 1787575633.457028 mapwrite 0x019018 thru 0x01d11e (0x4107 bytes) 000173 1787575633.481014 mapread 0x01f26d thru 0x0213c7 (0x215b bytes) 000174 1787575633.481806 write 0x014a68 thru 0x0241d7 (0xf770 bytes) 000175 1787575633.489738 write 0x033aa7 thru 0x03e287 (0xa7e1 bytes) 000176 1787575633.498899 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e288): Mirror 4: 0xd9490554 (stale) Mirror 6: 0xd9490554 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e288): Mirror 4: 0xd9490554 (stale) Mirror 6: 0xd9490554 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e288: skipped 000177 1787575633.836760 mapread 0x02410f thru 0x02ab6a (0x6a5c bytes) 000178 1787575633.880922 fallocate 0x018612 thru 0x023c0c (0xb5fb bytes) 000179 1787575633.896935 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e288): Mirror 4: 0xd9490554 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e288): Mirror 4: 0xd9490554 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e288: skipped 000180 1787575634.287196 trunc from 0x03b659 to 0x03e287 (0x2c2f bytes) 000181 1787575634.300023 fallocate 0x037391 thru 0x03e607 (0x7277 bytes) 000182 1787575634.317230 punch from 0x02dc27 to 0x03ae77 (0xd251 bytes) 000183 1787575634.324358 read 0x00dae0 thru 0x011fe9 (0x450a bytes) 000184 1787575634.335965 trunc from 0x038efb to 0x03e287 (0x538d bytes) 000185 1787575634.344606 read 0x01f197 thru 0x022c1e (0x3a88 bytes) 000186 1787575634.354587 trunc from 0x03e288 to 0x00fea9 (0x2e3de bytes) 000187 1787575634.370667 read 0x005611 thru 0x00a7a7 (0x5197 bytes) 000188 1787575634.371492 write 0x00a27b thru 0x01207e (0x7e04 bytes) 000189 1787575634.373411 mapwrite 0x012991 thru 0x01ece6 (0xc356 bytes) 000190 1787575634.446043 punch from 0x01e986 to 0x01ece6 (0x361 bytes) 000193 1787575634.454728 fallocate 0x028fe3 thru 0x02d883 (0x48a1 bytes) 000197 1787575634.469043 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d884): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d884): Mirror 4: 0x7af32419 (stale) CRC-32 checksum value for chunk [0, 0x2d884): Mirror 1: 0xc39fad53 Mirror 5: 0xc39fad53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d884: skipped 000198 1787575635.441536 read 0x00a5b8 thru 0x018fde (0xea27 bytes) 000199 1787575635.486824 write 0x008098 thru 0x008dee (0xd57 bytes) 000200 1787575635.570616 mapread 0x01140e thru 0x011c16 (0x809 bytes) 000201 1787575635.600178 MIRROR_RESYNC 3 mirrors 000202 1787575636.277442 trunc from 0x02d884 to 0x01525d (0x18626 bytes) 000203 1787575636.316980 trunc from 0x011072 to 0x013069 (0x1ff8 bytes) 000204 1787575636.324008 read 0x002c61 thru 0x00e8f6 (0xbc96 bytes) 000205 1787575636.334714 trunc from 0x011355 to 0x01525d (0x3f09 bytes) 000206 1787575636.340458 fallocate 0x011a72 thru 0x019121 (0x76b0 bytes) 000207 1787575636.344832 read 0x002aef thru 0x005d93 (0x32a5 bytes) 000208 1787575636.345447 write 0x02aad4 thru 0x0348ff (0x9e2c bytes) 000209 1787575636.347454 trunc from 0x01166c to 0x01634d (0x4ce2 bytes) 000210 1787575636.352497 read 0x0324d8 thru 0x0348ff (0x2428 bytes) 000211 1787575636.353232 read 0x01f306 thru 0x0236a3 (0x439e bytes) 000212 1787575636.361707 trunc from 0x034900 to 0x03fb7f (0xb280 bytes) 000213 1787575636.384352 mapread 0x006b4e thru 0x00a8c1 (0x3d74 bytes) 000214 1787575636.386526 trunc from 0x016cb5 to 0x023ae0 (0xce2c bytes) 000215 1787575636.392765 write 0x01b335 thru 0x020663 (0x532f bytes) 000218 1787575636.409164 fallocate 0x0188e9 thru 0x0279e4 (0xf0fc bytes) 000219 1787575636.415757 read 0x00c7bc thru 0x01b1da (0xea1f bytes) 000220 1787575636.434046 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb80): Mirror 5: 0x5b18bf39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb80): Mirror 5: 0x5b18bf39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb80: skipped 000221 1787575636.736769 trunc from 0x0055e5 to 0x0153f0 (0xfe0c bytes) 000222 1787575636.740644 mapwrite 0x0343ba thru 0x03ffff (0xbc46 bytes) 000223 1787575636.805522 write 0x001506 thru 0x004898 (0x3393 bytes) 000224 1787575636.818603 mapread 0x01db25 thru 0x0272f6 (0x97d2 bytes) 000226 1787575636.849283 trunc from 0x012ac1 to 0x01a08e (0x75ce bytes) 000227 1787575636.857316 read 0x039beb thru 0x03ffff (0x6415 bytes) 000228 1787575636.858884 trunc from 0x040000 to 0x023efa (0x1c105 bytes) 000229 1787575636.883104 fallocate 0x01dce8 thru 0x027187 (0x94a0 bytes) 000230 1787575636.890816 mapwrite 0x0034e7 thru 0x00ebbf (0xb6d9 bytes) 000231 1787575636.928342 write 0x008a0f thru 0x0123d1 (0x99c3 bytes) 000232 1787575636.950947 fallocate 0x00fc6c thru 0x0116c9 (0x1a5e bytes) 000233 1787575636.961615 trunc from 0x023efb to 0x03313d (0xf243 bytes) 000234 1787575636.983398 punch from 0x023b84 to 0x02b1f7 (0x7674 bytes) 000235 1787575636.990349 write 0x008913 thru 0x0152f2 (0xc9e0 bytes) 000236 1787575636.999568 mapwrite 0x01a719 thru 0x01ae2a (0x712 bytes) 000237 1787575637.034574 trunc from 0x01bba9 to 0x022efe (0x7356 bytes) 000238 1787575637.047610 mapread 0x0259a8 thru 0x02f4ee (0x9b47 bytes) 000239 1787575637.088975 mapread 0x01c6af thru 0x025dad (0x96ff bytes) 000240 1787575637.105509 mapread 0x0165fd thru 0x01e8ca (0x82ce bytes) 000241 1787575637.114908 fallocate 0x0020a9 thru 0x006aa6 (0x49fe bytes) 000242 1787575637.127947 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3313e): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3313e): Mirror 5: 0xd3e6d475 (stale) CRC-32 checksum value for chunk [0, 0x3313e): Mirror 1: 0xb997edce Mirror 6: 0xb997edce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3313e: skipped 000243 1787575638.257628 mapwrite 0x009743 thru 0x01490b (0xb1c9 bytes) 000244 1787575638.434947 punch from 0x03091b to 0x03313d (0x2823 bytes) 000246 1787575638.441337 punch from 0x01381c to 0x01df79 (0xa75e bytes) 000247 1787575638.447136 trunc from 0x03313e to 0x006f2c (0x2c211 bytes) 000248 1787575638.509684 write 0x014503 thru 0x022629 (0xe127 bytes) 000249 1787575638.511619 write 0x00ea34 thru 0x0155b7 (0x6b84 bytes) 000250 1787575638.518877 write 0x03310e thru 0x03ffff (0xcef2 bytes) 000251 1787575638.521904 write 0x036c1d thru 0x03d55c (0x6940 bytes) 000252 1787575638.530267 MIRROR_RESYNC 3 mirrors 000253 1787575639.620628 write 0x01172b thru 0x01a3e4 (0x8cba bytes) 000254 1787575639.710105 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xedad54b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xedad54b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000255 1787575640.087502 mapread 0x02bf57 thru 0x02fd5d (0x3e07 bytes) 000257 1787575640.111744 trunc from 0x008e23 to 0x0151d4 (0xc3b2 bytes) 000258 1787575640.119762 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xedad54b6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3d947b16 Mirror 7: 0x3d947b16 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000259 1787575640.927611 punch from 0x0018b9 to 0x00f528 (0xdc70 bytes) 000260 1787575640.971921 trunc from 0x040000 to 0x01fe07 (0x201f8 bytes) 000261 1787575641.009168 MIRROR_RESYNC 3 mirrors 000263 1787575642.086136 trunc from 0x00ac8b to 0x0178a2 (0xcc18 bytes) 000264 1787575642.125089 read 0x0193c8 thru 0x01fe07 (0x6a40 bytes) 000265 1787575642.136620 read 0x014e1d thru 0x01552b (0x70f bytes) 000266 1787575642.164857 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fe08): Mirror 7: 0x219e1d84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fe08): Mirror 7: 0x219e1d84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fe08: skipped 000267 1787575642.578870 read 0x00a5de thru 0x00adfb (0x81e bytes) 000268 1787575642.604647 mapwrite 0x017128 thru 0x0201dd (0x90b6 bytes) 000269 1787575642.733422 trunc from 0x0188fd to 0x01a39d (0x1aa1 bytes) 000270 1787575642.744684 punch from 0x00e39d to 0x017a3f (0x96a3 bytes) 000271 1787575642.754333 mapwrite 0x01f5fb thru 0x0284d4 (0x8eda bytes) 000272 1787575642.883860 mapwrite 0x019ac4 thru 0x01a95d (0xe9a bytes) 000273 1787575642.935253 mapread 0x018542 thru 0x022970 (0xa42f bytes) 000275 1787575642.936545 trunc from 0x0284d5 to 0x01c558 (0xbf7c bytes) 000276 1787575642.975625 MIRROR_RESYNC 2 mirrors 000277 1787575643.526470 trunc from 0x012e9c to 0x01c558 (0x96bd bytes) 000278 1787575643.574724 fallocate 0x034f36 thru 0x03dda6 (0x8e71 bytes) 000279 1787575643.585563 fallocate 0x03333d thru 0x0391bd (0x5e81 bytes) 000281 1787575643.595114 trunc from 0x0391be to 0x006452 (0x32d6b bytes) 000283 1787575643.630702 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x6453): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6453): Mirror 7: 0xe23f4adb (stale) CRC-32 checksum value for chunk [0, 0x6453): Mirror 1: 0xe23f4adb Mirror 8: 0xe23f4adb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6453: skipped 000284 1787575644.915614 fallocate 0x03ecd6 thru 0x03ffff (0x132a bytes) 000285 1787575644.964616 trunc from 0x025d4d to 0x028ad5 (0x2d89 bytes) 000286 1787575644.978258 write 0x015bb7 thru 0x01a168 (0x45b2 bytes) 000287 1787575645.036273 trunc from 0x010f9a to 0x013a3d (0x2aa4 bytes) 000288 1787575645.048571 trunc from 0x023f35 to 0x02674b (0x2817 bytes) 000289 1787575645.074733 punch from 0x03f491 to 0x03ffff (0xb6f bytes) 000290 1787575645.083292 trunc from 0x040000 to 0x00cf17 (0x330e8 bytes) 000291 1787575645.141909 mapread 0x002f4a thru 0x007289 (0x4340 bytes) 000293 1787575645.184951 mapwrite 0x0263ee thru 0x0304c2 (0xa0d5 bytes) 000294 1787575645.287581 trunc from 0x022f8e to 0x02707f (0x40f2 bytes) 000295 1787575645.298413 MIRROR_RESYNC 3 mirrors 000296 1787575646.550885 fallocate 0x01ba04 thru 0x026c0c (0xb209 bytes) 000297 1787575646.603724 punch from 0x012f97 to 0x022bc4 (0xfc2e bytes) 000298 1787575646.617269 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x304c3): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x304c3): Mirror 7: 0xaa9c5676 (stale) Mirror 8: 0xaa9c5676 (stale) CRC-32 checksum value for chunk [0, 0x304c3): Mirror 1: 0xaa9c5676 Mirror 9: 0xaa9c5676 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x304c3: skipped 000299 1787575647.659016 fallocate 0x039198 thru 0x03cd9b (0x3c04 bytes) 000300 1787575647.724035 mapread 0x01f970 thru 0x02a527 (0xabb8 bytes) 000301 1787575647.750387 fallocate 0x00c83b thru 0x00f9e3 (0x31a9 bytes) 000302 1787575647.755759 trunc from 0x01ca0f to 0x01f08b (0x267d bytes) 000303 1787575647.760441 write 0x013bff thru 0x019518 (0x591a bytes) 000304 1787575647.787323 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x304c3): Mirror 7: 0xaa9c5676 (stale) Mirror 8: 0xaa9c5676 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x304c3): Mirror 7: 0xaa9c5676 (stale) Mirror 8: 0xaa9c5676 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x304c3: skipped 000305 1787575648.109502 fallocate 0x034ed0 thru 0x03da97 (0x8bc8 bytes) 000306 1787575648.114462 fallocate 0x02306c thru 0x02e1f7 (0xb18c bytes) 000307 1787575648.117767 read 0x0193db thru 0x026625 (0xd24b bytes) 000308 1787575648.126686 mapread 0x017b0f thru 0x01e2c6 (0x67b8 bytes) 000309 1787575648.134028 write 0x02a013 thru 0x0383d1 (0xe3bf bytes) 000310 1787575648.145038 trunc from 0x0383d2 to 0x02cc40 (0xb791 bytes) 000311 1787575648.155059 punch from 0x00dffb to 0x0115ef (0x35f5 bytes) 000312 1787575648.159562 trunc from 0x023ea2 to 0x028221 (0x4380 bytes) 000313 1787575648.163233 trunc from 0x01f808 to 0x0254d6 (0x5ccf bytes) 000314 1787575648.167253 mapwrite 0x02c181 thru 0x02d3b8 (0x1238 bytes) 000315 1787575648.208701 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d3b9): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d3b9): Mirror 7: 0xc7b2ad53 (stale) Mirror 8: 0xc7b2ad53 (stale) CRC-32 checksum value for chunk [0, 0x2d3b9): Mirror 1: 0x855c7ddb Mirror 9: 0x855c7ddb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d3b9: skipped 000316 1787575649.130521 write 0x03c914 thru 0x03f1c5 (0x28b2 bytes) 000317 1787575649.189576 trunc from 0x0185ea to 0x0250ab (0xcac2 bytes) 000318 1787575649.197350 read 0x03a815 thru 0x03d598 (0x2d84 bytes) 000319 1787575649.203853 write 0x01515e thru 0x019b83 (0x4a26 bytes) 000320 1787575649.223419 trunc from 0x03f1c6 to 0x0340f5 (0xb0d0 bytes) 000321 1787575649.286727 trunc from 0x012d6f to 0x014e40 (0x20d2 bytes) 000322 1787575649.294741 mapwrite 0x03968b thru 0x03f381 (0x5cf7 bytes) 000323 1787575649.384730 trunc from 0x0174fd to 0x019083 (0x1b87 bytes) 000324 1787575649.391819 mapwrite 0x029ffd thru 0x0332d2 (0x92d6 bytes) 000327 1787575649.435636 fallocate 0x03ac4e thru 0x03c93d (0x1cf0 bytes) 000328 1787575649.444908 read 0x02e7ff thru 0x0349f9 (0x61fb bytes) 000329 1787575649.451687 MIRROR_RESYNC 4 mirrors 000330 1787575651.098620 read 0x03a685 thru 0x03f381 (0x4cfd bytes) 000331 1787575651.115337 trunc from 0x038580 to 0x03f381 (0x6e02 bytes) 000332 1787575651.171483 trunc from 0x03f382 to 0x02b508 (0x13e79 bytes) 000333 1787575651.209220 fallocate 0x00e5ca thru 0x018480 (0x9eb7 bytes) 000334 1787575651.220010 trunc from 0x003c4f to 0x006732 (0x2ae4 bytes) 000335 1787575651.231364 read 0x01cd03 thru 0x01d559 (0x857 bytes) 000336 1787575651.246559 trunc from 0x028bea to 0x02b508 (0x291f bytes) 000337 1787575651.253527 mapwrite 0x012a12 thru 0x013f88 (0x1577 bytes) 000339 1787575651.275592 mapwrite 0x01df33 thru 0x02cf42 (0xf010 bytes) 000340 1787575651.336720 trunc from 0x02cf43 to 0x038516 (0xb5d4 bytes) 000341 1787575651.375431 mapread 0x00120c thru 0x010c08 (0xf9fd bytes) 000343 1787575651.389403 fallocate 0x03eff7 thru 0x03ffff (0x1009 bytes) 000344 1787575651.399759 fallocate 0x03a4d7 thru 0x03ffff (0x5b29 bytes) 000345 1787575651.407576 mapread 0x0216ef thru 0x0229ba (0x12cc bytes) 000346 1787575651.408398 read 0x0258df thru 0x031c53 (0xc375 bytes) 000347 1787575651.416374 read 0x01502a thru 0x01f0eb (0xa0c2 bytes) 000348 1787575651.417136 mapwrite 0x039166 thru 0x03ffff (0x6e9a bytes) 000349 1787575651.472410 mapread 0x0147f7 thru 0x0153a4 (0xbae bytes) 000350 1787575651.473279 trunc from 0x02fe30 to 0x030cda (0xeab bytes) 000351 1787575651.483109 punch from 0x022d18 to 0x028edd (0x61c6 bytes) 000352 1787575651.495583 mapwrite 0x032060 thru 0x0381dc (0x617d bytes) 000353 1787575651.515505 fallocate 0x03d2bc thru 0x03ffff (0x2d44 bytes) 000354 1787575651.520177 mapwrite 0x02e4ae thru 0x03d010 (0xeb63 bytes) 000355 1787575651.566154 mapread 0x00b31a thru 0x0115ca (0x62b1 bytes) 000356 1787575651.567596 punch from 0x023ec2 to 0x0291e2 (0x5321 bytes) 000357 1787575651.573699 fallocate 0x02af0b thru 0x02ccdb (0x1dd1 bytes) 000358 1787575651.586990 punch from 0x005583 to 0x008558 (0x2fd6 bytes) 000359 1787575651.607405 mapwrite 0x030de3 thru 0x03e70c (0xd92a bytes) 000360 1787575651.632963 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9ae1e360 (stale) Mirror 8: 0x9ae1e360 (stale) Mirror 9: 0x9ae1e360 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfecf1311 Mirror 10: 0xfecf1311 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000361 1787575653.074568 read 0x03a2bd thru 0x03f026 (0x4d6a bytes) 000362 1787575653.104049 write 0x01bb0c thru 0x026f49 (0xb43e bytes) 000364 1787575653.188512 write 0x03bb9f thru 0x03ffff (0x4461 bytes) 000365 1787575653.194196 mapwrite 0x00f1d9 thru 0x00f5e2 (0x40a bytes) 000366 1787575653.220927 punch from 0x03ab9f to 0x03b11a (0x57c bytes) 000367 1787575653.229343 punch from 0x028ecc to 0x02ac7f (0x1db4 bytes) 000369 1787575653.250264 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9ae1e360 (stale) Mirror 8: 0x9ae1e360 (stale) Mirror 9: 0x9ae1e360 (stale) Mirror 10: 0xfecf1311 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x979c035b Mirror 11: 0x979c035b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000371 1787575654.506396 fallocate 0x004484 thru 0x008cae (0x482b bytes) 000372 1787575654.604501 fallocate 0x003d2e thru 0x00e1a5 (0xa478 bytes) 000373 1787575654.617132 write 0x03a5b6 thru 0x03ffff (0x5a4a bytes) 000374 1787575654.641437 read 0x00cf4d thru 0x00eeaa (0x1f5e bytes) 000376 1787575654.649118 fallocate 0x00a8cf thru 0x019f0a (0xf63c bytes) 000377 1787575654.658530 fallocate 0x03359b thru 0x037ce1 (0x4747 bytes) 000378 1787575654.664035 read 0x02b676 thru 0x0360da (0xaa65 bytes) 000379 1787575654.679451 read 0x0302ff thru 0x039993 (0x9695 bytes) 000380 1787575654.680714 write 0x011915 thru 0x01a897 (0x8f83 bytes) 000381 1787575654.683451 mapwrite 0x00db54 thru 0x01410d (0x65ba bytes) 000382 1787575654.707641 write 0x024419 thru 0x028b3f (0x4727 bytes) 000383 1787575654.709294 mapwrite 0x02ef8c thru 0x0376b8 (0x872d bytes) 000384 1787575654.753638 mapread 0x026fd5 thru 0x028a23 (0x1a4f bytes) 000385 1787575654.762715 MIRROR_RESYNC 6 mirrors 000386 1787575657.426525 mapread 0x029e64 thru 0x032261 (0x83fe bytes) 000387 1787575657.471378 mapread 0x039bcb thru 0x03ffff (0x6435 bytes) 000389 1787575657.493566 mapread 0x034d41 thru 0x03cf47 (0x8207 bytes) 000390 1787575657.495552 trunc from 0x040000 to 0x02eed1 (0x1112e bytes) 000391 1787575657.543911 read 0x01fc13 thru 0x02bddd (0xc1cb bytes) 000392 1787575657.559382 mapwrite 0x027d8b thru 0x036c52 (0xeec8 bytes) 000394 1787575657.610355 write 0x0348ae thru 0x03b97b (0x70ce bytes) 000395 1787575657.612269 mapread 0x02a2a3 thru 0x0386e6 (0xe444 bytes) 000396 1787575657.617047 MIRROR_RESYNC 6 mirrors 000398 1787575659.049419 trunc from 0x03b97c to 0x031073 (0xa908 bytes) 000399 1787575659.190766 punch from 0x002c36 to 0x004b87 (0x1f52 bytes) 000400 1787575659.214193 fallocate 0x030984 thru 0x03b911 (0xaf8e bytes) 000401 1787575659.233485 write 0x000d80 thru 0x00e1c3 (0xd444 bytes) 000402 1787575659.260722 fallocate 0x01ed2e thru 0x029c13 (0xaee6 bytes) 000403 1787575659.284201 punch from 0x01594c to 0x02263f (0xccf4 bytes) 000405 1787575659.303809 trunc from 0x00cdb5 to 0x01be36 (0xf082 bytes) 000406 1787575659.355173 write 0x02c3a8 thru 0x035cdf (0x9938 bytes) 000407 1787575659.381477 trunc from 0x009532 to 0x00b2b7 (0x1d86 bytes) 000408 1787575659.404468 read 0x00095d thru 0x00f7b0 (0xee54 bytes) 000409 1787575659.413875 fallocate 0x03d5d1 thru 0x03ffff (0x2a2f bytes) 000410 1787575659.442500 mapwrite 0x039af1 thru 0x03ffff (0x650f bytes) 000411 1787575659.616188 write 0x037075 thru 0x03ffff (0x8f8b bytes) 000412 1787575659.621409 write 0x0057a9 thru 0x012cd7 (0xd52f bytes) 000413 1787575659.658326 punch from 0x03019d to 0x03cd7e (0xcbe2 bytes) 000414 1787575659.709718 punch from 0x011d7f to 0x017ec2 (0x6144 bytes) 000415 1787575659.734839 read 0x01f8b3 thru 0x0252a2 (0x59f0 bytes) 000416 1787575659.755410 write 0x0048af thru 0x00f96b (0xb0bd bytes) 000417 1787575659.760882 trunc from 0x01fec9 to 0x023ecf (0x4007 bytes) 000418 1787575659.778310 read 0x01843a thru 0x01fc3c (0x7803 bytes) 000419 1787575659.823868 punch from 0x00100a to 0x00cd59 (0xbd50 bytes) 000421 1787575659.893324 mapwrite 0x001956 thru 0x0112e8 (0xf993 bytes) 000422 1787575660.045108 read 0x026b24 thru 0x02d04b (0x6528 bytes) 000423 1787575660.046312 trunc from 0x027999 to 0x02c9a3 (0x500b bytes) 000424 1787575660.059794 fallocate 0x007946 thru 0x012cf7 (0xb3b2 bytes) 000426 1787575660.073994 write 0x01588f thru 0x01a60c (0x4d7e bytes) 000427 1787575660.092289 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd8ce4d8e (stale) Mirror 9: 0xd8ce4d8e (stale) Mirror 10: 0xd8ce4d8e (stale) Mirror 11: 0xd8ce4d8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xd8ce4d8e (stale) Mirror 9: 0xd8ce4d8e (stale) Mirror 10: 0xd8ce4d8e (stale) Mirror 11: 0xd8ce4d8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000428 1787575660.794321 mapwrite 0x025248 thru 0x030bdb (0xb994 bytes) 000429 1787575660.868467 trunc from 0x03d96e to 0x03f729 (0x1dbc bytes) 000430 1787575660.874924 punch from 0x012dcb to 0x0205b1 (0xd7e7 bytes) 000431 1787575660.886451 mapread 0x010983 thru 0x0111ea (0x868 bytes) 000432 1787575660.920225 trunc from 0x038faf to 0x03dc9f (0x4cf1 bytes) 000435 1787575660.938726 write 0x01e83a thru 0x029372 (0xab39 bytes) 000437 1787575660.952442 fallocate 0x00cb19 thru 0x011e20 (0x5308 bytes) 000438 1787575660.958995 punch from 0x02c44d to 0x032943 (0x64f7 bytes) 000439 1787575660.984722 mapwrite 0x014569 thru 0x015a7c (0x1514 bytes) 000440 1787575661.018931 trunc from 0x040000 to 0x01bd1a (0x242e5 bytes) 000441 1787575661.056430 trunc from 0x01bd1b to 0x039c09 (0x1deef bytes) 000442 1787575661.079253 mapwrite 0x02f934 thru 0x03021e (0x8eb bytes) 000443 1787575661.140262 mapread 0x010492 thru 0x01fe0f (0xf97e bytes) 000444 1787575661.172020 trunc from 0x039c0a to 0x026244 (0x139c5 bytes) 000445 1787575661.220442 mapread 0x000fd7 thru 0x006d89 (0x5db3 bytes) 000446 1787575661.240149 trunc from 0x026245 to 0x02415f (0x20e5 bytes) 000448 1787575661.263342 read 0x01a6fa thru 0x02415f (0x9a66 bytes) 000449 1787575661.269038 read 0x007651 thru 0x014114 (0xcac4 bytes) 000452 1787575661.270611 read 0x000b18 thru 0x0049fe (0x3ee7 bytes) 000453 1787575661.272821 punch from 0x003778 to 0x005cb8 (0x2541 bytes) 000454 1787575661.290257 mapread 0x00e831 thru 0x01ac5b (0xc42b bytes) 000455 1787575661.292122 punch from 0x01e14a to 0x02415f (0x6016 bytes) 000456 1787575661.310955 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24160): Mirror 9: 0xa68aef9 (stale) Mirror 10: 0xa68aef9 (stale) Mirror 11: 0xa68aef9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24160): Mirror 9: 0xa68aef9 (stale) Mirror 10: 0xa68aef9 (stale) Mirror 11: 0xa68aef9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24160: skipped 000458 1787575661.966444 mapwrite 0x034a98 thru 0x036d78 (0x22e1 bytes) 000459 1787575662.062362 trunc from 0x036d79 to 0x017f73 (0x1ee05 bytes) 000460 1787575662.117901 read 0x00eff7 thru 0x0177bd (0x87c7 bytes) 000461 1787575662.128890 fallocate 0x03c9fa thru 0x03ffff (0x3606 bytes) 000462 1787575662.139004 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x17f74): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17f74): Mirror 9: 0xea910087 (stale) Mirror 10: 0xea910087 (stale) Mirror 11: 0xea910087 (stale) CRC-32 checksum value for chunk [0, 0x17f74): Mirror 1: 0xbff43285 Mirror 12: 0xbff43285 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17f74: skipped 000463 1787575663.191254 punch from 0x0122d2 to 0x017f73 (0x5ca2 bytes) 000464 1787575663.250449 MIRROR_RESYNC 5 mirrors 000465 1787575664.836445 mapwrite 0x008422 thru 0x013b05 (0xb6e4 bytes) 000466 1787575665.046532 mapread 0x0013f3 thru 0x00c14f (0xad5d bytes) 000467 1787575665.060248 mapwrite 0x0296bf thru 0x02ccda (0x361c bytes) 000468 1787575665.217326 trunc from 0x019f11 to 0x0226cd (0x87bd bytes) 000469 1787575665.251718 MIRROR_RESYNC 5 mirrors 000470 1787575667.221403 read 0x01ff1a thru 0x02154b (0x1632 bytes) 000471 1787575667.227661 trunc from 0x02ccdb to 0x025315 (0x79c5 bytes) 000472 1787575667.264746 read 0x007b31 thru 0x00b43e (0x390e bytes) 000473 1787575667.274612 punch from 0x0202e2 to 0x025315 (0x5034 bytes) 000474 1787575667.283689 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25316): Mirror 9: 0xd6b0674c (stale) Mirror 11: 0xd6b0674c (stale) Mirror 12: 0xd6b0674c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25316): Mirror 9: 0xd6b0674c (stale) Mirror 11: 0xd6b0674c (stale) Mirror 12: 0xd6b0674c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25316: skipped 000475 1787575667.730475 trunc from 0x022be9 to 0x025315 (0x272d bytes) 000476 1787575667.738644 write 0x007840 thru 0x00ee75 (0x7636 bytes) 000477 1787575667.776992 write 0x00aca6 thru 0x016586 (0xb8e1 bytes) 000478 1787575667.792023 read 0x00b8a5 thru 0x018e8d (0xd5e9 bytes) 000479 1787575667.804710 read 0x0045aa thru 0x00ea06 (0xa45d bytes) 000483 1787575667.826132 read 0x00408b thru 0x00763c (0x35b2 bytes) 000484 1787575667.828931 write 0x03de1b thru 0x03ffff (0x21e5 bytes) 000485 1787575667.832181 read 0x030087 thru 0x03ea13 (0xe98d bytes) 000486 1787575667.847981 write 0x01e3f7 thru 0x020587 (0x2191 bytes) 000487 1787575667.849618 trunc from 0x00d4e1 to 0x010541 (0x3061 bytes) 000488 1787575667.886755 fallocate 0x001fb0 thru 0x00b9d2 (0x9a23 bytes) 000489 1787575667.897554 trunc from 0x040000 to 0x02e0d5 (0x11f2a bytes) 000490 1787575667.914224 read 0x012084 thru 0x01e666 (0xc5e3 bytes) 000491 1787575667.915178 mapwrite 0x01ee56 thru 0x024c90 (0x5e3b bytes) 000492 1787575667.971616 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e0d6): Mirror 9: 0x7aeefdc3 (stale) Mirror 12: 0x7aeefdc3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0d6): Mirror 9: 0x7aeefdc3 (stale) Mirror 12: 0x7aeefdc3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0d6: skipped 000493 1787575668.309667 trunc from 0x02e0d6 to 0x0065ea (0x27aeb bytes) 000494 1787575668.335598 mapread 0x0019d7 thru 0x004950 (0x2f7a bytes) 000495 1787575668.351507 trunc from 0x0065eb to 0x005bbd (0xa2d bytes) 000496 1787575668.366842 mapread 0x000098 thru 0x005bbd (0x5b26 bytes) 000497 1787575668.377733 MIRROR_RESYNC 3 mirrors 000498 1787575669.298873 write 0x020b3b thru 0x02fb19 (0xefdf bytes) 000500 1787575669.378527 punch from 0x02f504 to 0x02fb19 (0x616 bytes) 000501 1787575669.429350 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2fb1a): Mirror 9: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fb1a): Mirror 9: 0x35da5f21 (stale) Mirror 12: 0x35da5f21 (stale) CRC-32 checksum value for chunk [0, 0x2fb1a): Mirror 1: 0xe73134c7 Mirror 13: 0xe73134c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fb1a: skipped 000502 1787575670.481178 mapread 0x025c7c thru 0x02b26a (0x55ef bytes) 000503 1787575670.512002 trunc from 0x001d30 to 0x00a244 (0x8515 bytes) 000504 1787575670.559181 write 0x003577 thru 0x0122a2 (0xed2c bytes) 000505 1787575670.578300 write 0x03bc38 thru 0x03ffff (0x43c8 bytes) 000506 1787575670.580350 fallocate 0x02ccd7 thru 0x02f3ed (0x2717 bytes) 000507 1787575670.585299 mapread 0x01f711 thru 0x01feba (0x7aa bytes) 000508 1787575670.596059 trunc from 0x009699 to 0x00d337 (0x3c9f bytes) 000509 1787575670.613247 mapwrite 0x00951b thru 0x014b4c (0xb632 bytes) 000510 1787575670.655821 trunc from 0x040000 to 0x02ff40 (0x100bf bytes) 000511 1787575670.681433 trunc from 0x019e65 to 0x023600 (0x979c bytes) 000512 1787575670.691888 read 0x01cac0 thru 0x02168f (0x4bd0 bytes) 000513 1787575670.699217 mapread 0x00e846 thru 0x013e86 (0x5641 bytes) 000514 1787575670.700217 mapread 0x00634d thru 0x00e4a7 (0x815b bytes) 000515 1787575670.702103 punch from 0x02c3ee to 0x02ff40 (0x3b53 bytes) 000516 1787575670.707056 trunc from 0x02ff41 to 0x03269c (0x275c bytes) 000517 1787575670.722053 mapread 0x01efc5 thru 0x024955 (0x5991 bytes) 000518 1787575670.733106 MIRROR_RESYNC 4 mirrors 000519 1787575671.773172 mapread 0x00a87a thru 0x00cec5 (0x264c bytes) 000521 1787575671.790659 trunc from 0x03269d to 0x0348f6 (0x225a bytes) 000522 1787575671.833966 punch from 0x01fbd7 to 0x02a297 (0xa6c1 bytes) 000523 1787575671.841672 punch from 0x0036cc to 0x00e204 (0xab39 bytes) 000524 1787575671.847170 mapwrite 0x00372f thru 0x00f13a (0xba0c bytes) 000525 1787575671.875680 trunc from 0x0348f7 to 0x01d7a1 (0x17155 bytes) 000526 1787575671.897571 write 0x02dfed thru 0x03a6fc (0xc710 bytes) 000527 1787575671.899635 read 0x01642f thru 0x0249f4 (0xe5c6 bytes) 000528 1787575671.911092 trunc from 0x03a6fd to 0x007383 (0x33379 bytes) 000529 1787575671.924115 read 0x0071cc thru 0x007383 (0x1b8 bytes) 000530 1787575671.924850 trunc from 0x00140c to 0x007383 (0x5f78 bytes) 000531 1787575671.930769 punch from 0x000e2b to 0x007383 (0x6559 bytes) 000532 1787575671.937767 mapread 0x004049 thru 0x007383 (0x333b bytes) 000533 1787575671.948984 read 0x0042ec thru 0x007383 (0x3098 bytes) 000534 1787575671.949685 mapread 0x000466 thru 0x005845 (0x53e0 bytes) 000535 1787575671.955060 MIRROR_RESYNC 4 mirrors 000536 1787575672.877561 read 0x00662f thru 0x007383 (0xd55 bytes) 000537 1787575672.885039 read 0x0038ba thru 0x007383 (0x3aca bytes) 000538 1787575672.895285 punch from 0x001af0 to 0x0067a6 (0x4cb7 bytes) 000539 1787575672.926525 trunc from 0x00585c to 0x007383 (0x1b28 bytes) 000540 1787575672.932214 trunc from 0x001fc9 to 0x007383 (0x53bb bytes) 000541 1787575672.936748 trunc from 0x000f6d to 0x007383 (0x6417 bytes) 000543 1787575672.944582 fallocate 0x02e931 thru 0x032217 (0x38e7 bytes) 000544 1787575672.950220 read 0x002462 thru 0x002f75 (0xb14 bytes) 000545 1787575672.958712 write 0x01ff83 thru 0x02f939 (0xf9b7 bytes) 000546 1787575672.961762 fallocate 0x010000 thru 0x015b4f (0x5b50 bytes) 000547 1787575672.967423 write 0x03703a thru 0x03ffff (0x8fc6 bytes) 000548 1787575672.970280 write 0x03ffe6 thru 0x03ffff (0x1a bytes) 000550 1787575672.972242 mapwrite 0x037412 thru 0x03b3bb (0x3faa bytes) 000551 1787575673.002446 MIRROR_RESYNC 4 mirrors 000552 1787575674.075635 punch from 0x01de3e to 0x022a37 (0x4bfa bytes) 000553 1787575674.134702 mapread 0x03c54d thru 0x03ffff (0x3ab3 bytes) 000554 1787575674.172729 read 0x01effa thru 0x0283bf (0x93c6 bytes) 000555 1787575674.187146 trunc from 0x0299ce to 0x03166b (0x7c9e bytes) 000556 1787575674.200542 read 0x0284ce thru 0x030244 (0x7d77 bytes) 000557 1787575674.212673 fallocate 0x025d5f thru 0x02a251 (0x44f3 bytes) 000558 1787575674.218278 mapread 0x033025 thru 0x03739c (0x4378 bytes) 000559 1787575674.219309 punch from 0x001f2e to 0x00379e (0x1871 bytes) 000560 1787575674.224472 trunc from 0x018d70 to 0x02728d (0xe51e bytes) 000562 1787575674.230574 trunc from 0x040000 to 0x018325 (0x27cda bytes) 000563 1787575674.253113 write 0x00a951 thru 0x016b89 (0xc239 bytes) 000564 1787575674.258466 mapread 0x001804 thru 0x003305 (0x1b02 bytes) 000566 1787575674.273567 fallocate 0x00ebb4 thru 0x011645 (0x2a92 bytes) 000569 1787575674.280643 mapread 0x013336 thru 0x01575f (0x242a bytes) 000570 1787575674.282022 mapwrite 0x024195 thru 0x0242e0 (0x14c bytes) 000571 1787575674.318300 mapread 0x00ab5e thru 0x00cfb1 (0x2454 bytes) 000572 1787575674.319619 punch from 0x01dc8e to 0x0242e0 (0x6653 bytes) 000573 1787575674.325560 fallocate 0x00944f thru 0x00b615 (0x21c7 bytes) 000574 1787575674.331156 trunc from 0x00c28a to 0x016c8d (0xaa04 bytes) 000575 1787575674.336677 mapread 0x01a77b thru 0x01e63c (0x3ec2 bytes) 000576 1787575674.365619 punch from 0x00c36a to 0x019e49 (0xdae0 bytes) 000577 1787575674.370882 punch from 0x012cb4 to 0x020508 (0xd855 bytes) 000578 1787575674.375126 punch from 0x004e6c to 0x00c720 (0x78b5 bytes) 000579 1787575674.380518 mapread 0x020ecc thru 0x0242e0 (0x3415 bytes) 000581 1787575674.406004 punch from 0x01f56a to 0x0242e0 (0x4d77 bytes) 000582 1787575674.412159 trunc from 0x0242e1 to 0x00549e (0x1ee42 bytes) 000583 1787575674.428169 trunc from 0x00549f to 0x023d43 (0x1e8a5 bytes) 000584 1787575674.442484 mapread 0x002392 thru 0x011a1d (0xf68c bytes) 000585 1787575674.535838 trunc from 0x0094d6 to 0x0157bd (0xc2e8 bytes) 000587 1787575674.547352 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x23d44): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23d44): Mirror 9: 0x2f7bd8a0 (stale) Mirror 12: 0x2f7bd8a0 (stale) Mirror 13: 0x2f7bd8a0 (stale) CRC-32 checksum value for chunk [0, 0x23d44): Mirror 1: 0x4bf6c856 Mirror 14: 0x4bf6c856 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23d44: skipped 000588 1787575676.006546 read 0x01a261 thru 0x023d43 (0x9ae3 bytes) 000589 1787575676.021137 trunc from 0x00e5fd to 0x01106f (0x2a73 bytes) 000590 1787575676.056135 mapread 0x0168bc thru 0x01ddea (0x752f bytes) 000591 1787575676.076645 trunc from 0x023d44 to 0x027d40 (0x3ffd bytes) 000593 1787575676.102371 read 0x003f24 thru 0x0094d0 (0x55ad bytes) 000594 1787575676.113691 trunc from 0x027d41 to 0x0023f6 (0x2594a bytes) 000595 1787575676.133393 mapread 0x0003a1 thru 0x0023f6 (0x2056 bytes) 000596 1787575676.137544 punch from 0x00215f to 0x0023f6 (0x298 bytes) 000597 1787575676.148032 read 0x001d5e thru 0x0023f6 (0x699 bytes) 000599 1787575676.154913 read 0x000a3b thru 0x0023f6 (0x19bc bytes) 000600 1787575676.155729 fallocate 0x000181 thru 0x00afcc (0xae4c bytes) 000601 1787575676.165265 MIRROR_RESYNC 5 mirrors 000602 1787575676.995993 write 0x0258db thru 0x033258 (0xd97e bytes) 000603 1787575677.030650 mapread 0x02d082 thru 0x033258 (0x61d7 bytes) 000604 1787575677.032041 write 0x030ccc thru 0x034412 (0x3747 bytes) 000605 1787575677.034002 mapread 0x002f13 thru 0x010e38 (0xdf26 bytes) 000606 1787575677.060021 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34413): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34413): Mirror 9: 0x2a272f85 (stale) Mirror 12: 0x2a272f85 (stale) Mirror 13: 0x2a272f85 (stale) Mirror 14: 0x2a272f85 (stale) CRC-32 checksum value for chunk [0, 0x34413): Mirror 1: 0x82255d9a Mirror 15: 0x82255d9a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34413: skipped 000607 1787575678.165896 mapread 0x02ea00 thru 0x034412 (0x5a13 bytes) 000608 1787575678.206965 mapwrite 0x03047c thru 0x034e57 (0x49dc bytes) 000609 1787575678.298062 trunc from 0x02d949 to 0x02f31e (0x19d6 bytes) 000610 1787575678.303352 read 0x01abfa thru 0x01e15a (0x3561 bytes) 000611 1787575678.310331 fallocate 0x032db7 thru 0x03c72d (0x9977 bytes) 000613 1787575678.315600 punch from 0x01f0e1 to 0x0203db (0x12fb bytes) 000614 1787575678.319816 mapread 0x02ac26 thru 0x034e57 (0xa232 bytes) 000615 1787575678.328594 punch from 0x00c7f0 to 0x011bf5 (0x5406 bytes) 000616 1787575678.333167 punch from 0x00deec to 0x0143bf (0x64d4 bytes) 000619 1787575678.338656 mapread 0x00a033 thru 0x0175d8 (0xd5a6 bytes) 000621 1787575678.357373 read 0x01a727 thru 0x01c0c7 (0x19a1 bytes) 000622 1787575678.358047 punch from 0x01a6da to 0x021c78 (0x759f bytes) 000623 1787575678.362352 mapwrite 0x021197 thru 0x02c0cb (0xaf35 bytes) 000624 1787575678.395169 MIRROR_RESYNC 6 mirrors 000625 1787575679.803839 fallocate 0x0359a9 thru 0x03ffff (0xa657 bytes) 000627 1787575679.872508 mapread 0x01bb4a thru 0x028dc5 (0xd27c bytes) 000628 1787575679.904686 mapread 0x02d4c3 thru 0x0344ec (0x702a bytes) 000629 1787575679.920006 punch from 0x016c4b to 0x021a46 (0xadfc bytes) 000630 1787575679.924718 mapwrite 0x004227 thru 0x00de76 (0x9c50 bytes) 000631 1787575679.950747 punch from 0x014af8 to 0x01f841 (0xad4a bytes) 000632 1787575679.959193 MIRROR_RESYNC 6 mirrors 000633 1787575681.688346 trunc from 0x037388 to 0x03ffff (0x8c78 bytes) 000634 1787575681.753305 trunc from 0x00eac6 to 0x01a603 (0xbb3e bytes) 000635 1787575681.763538 mapwrite 0x0149a7 thru 0x014ed5 (0x52f bytes) 000636 1787575681.796554 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb0ff82cc (stale) Mirror 12: 0xb0ff82cc (stale) Mirror 13: 0xb0ff82cc (stale) Mirror 14: 0xb0ff82cc (stale) Mirror 15: 0xb0ff82cc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5d483fd4 Mirror 16: 0x5d483fd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000637 1787575683.172200 punch from 0x01394c to 0x01476c (0xe21 bytes) 000638 1787575683.238769 mapread 0x031e1f thru 0x03262b (0x80d bytes) 000639 1787575683.253965 write 0x039d26 thru 0x03ffff (0x62da bytes) 000640 1787575683.264947 fallocate 0x038416 thru 0x03ffff (0x7bea bytes) 000641 1787575683.271302 trunc from 0x040000 to 0x01b0a9 (0x24f56 bytes) 000642 1787575683.286730 punch from 0x00c312 to 0x00f621 (0x3310 bytes) 000644 1787575683.291154 mapwrite 0x00c4ec thru 0x018b2c (0xc641 bytes) 000646 1787575683.340636 mapwrite 0x025b73 thru 0x02b0c6 (0x5554 bytes) 000647 1787575683.417914 read 0x00a53b thru 0x01985e (0xf324 bytes) 000648 1787575683.418637 trunc from 0x02b0c7 to 0x03a6e1 (0xf61b bytes) 000649 1787575683.443753 trunc from 0x030141 to 0x03a4ec (0xa3ac bytes) 000650 1787575683.448753 mapwrite 0x03aa0b thru 0x03ffff (0x55f5 bytes) 000651 1787575683.509678 fallocate 0x0352ca thru 0x03ffff (0xad36 bytes) 000652 1787575683.520924 trunc from 0x0381bc to 0x03d39c (0x51e1 bytes) 000654 1787575683.533298 trunc from 0x01a97c to 0x01fa55 (0x50da bytes) 000655 1787575683.548261 fallocate 0x0022e1 thru 0x00557a (0x329a bytes) 000656 1787575683.572491 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb0ff82cc (stale) Mirror 13: 0xb0ff82cc (stale) Mirror 14: 0xb0ff82cc (stale) Mirror 15: 0xb0ff82cc (stale) Mirror 16: 0x5d483fd4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xb0ff82cc (stale) Mirror 13: 0xb0ff82cc (stale) Mirror 14: 0xb0ff82cc (stale) Mirror 15: 0xb0ff82cc (stale) Mirror 16: 0x5d483fd4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000657 1787575684.201410 read 0x032131 thru 0x03ecfa (0xcbca bytes) 000658 1787575684.220251 trunc from 0x01afbd to 0x02a42d (0xf471 bytes) 000659 1787575684.228908 punch from 0x02f5bc to 0x0351c9 (0x5c0e bytes) 000660 1787575684.235056 trunc from 0x009c73 to 0x015bc5 (0xbf53 bytes) 000661 1787575684.239911 fallocate 0x02e3d9 thru 0x03182f (0x3457 bytes) 000662 1787575684.245037 mapwrite 0x021266 thru 0x02ebd8 (0xd973 bytes) 000663 1787575684.297073 mapwrite 0x022ef4 thru 0x026c96 (0x3da3 bytes) 000664 1787575684.326895 mapread 0x032416 thru 0x03ffff (0xdbea bytes) 000665 1787575684.338215 write 0x00df6b thru 0x01bf32 (0xdfc8 bytes) 000666 1787575684.357455 mapread 0x00b50e thru 0x00ec75 (0x3768 bytes) 000667 1787575684.373302 MIRROR_RESYNC 6 mirrors 000669 1787575686.744391 punch from 0x033b49 to 0x0372f4 (0x37ac bytes) 000670 1787575686.796397 punch from 0x0060f2 to 0x011075 (0xaf84 bytes) 000671 1787575686.803509 write 0x00415d thru 0x004211 (0xb5 bytes) 000673 1787575686.811227 trunc from 0x03b148 to 0x03d213 (0x20cc bytes) 000674 1787575686.817541 mapwrite 0x0233e8 thru 0x025586 (0x219f bytes) 000675 1787575686.877685 punch from 0x001833 to 0x006c99 (0x5467 bytes) 000676 1787575686.885339 mapread 0x0295a6 thru 0x0315b0 (0x800b bytes) 000677 1787575686.900645 trunc from 0x01cf50 to 0x023b42 (0x6bf3 bytes) 000678 1787575686.907273 punch from 0x001649 to 0x0106fb (0xf0b3 bytes) 000679 1787575686.912189 trunc from 0x03e89e to 0x03ffff (0x1762 bytes) 000680 1787575686.916923 write 0x00a9dd thru 0x00ba6b (0x108f bytes) 000681 1787575686.927616 mapwrite 0x01397c thru 0x01848d (0x4b12 bytes) 000682 1787575686.959417 punch from 0x02d31e to 0x02e07f (0xd62 bytes) 000683 1787575686.965224 punch from 0x031b46 to 0x03340b (0x18c6 bytes) 000684 1787575686.970448 punch from 0x035c4d to 0x03fcb6 (0xa06a bytes) 000685 1787575686.975002 mapwrite 0x0366db thru 0x03d495 (0x6dbb bytes) 000686 1787575687.012769 read 0x01c82e thru 0x01f86e (0x3041 bytes) 000687 1787575687.026242 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xbedbd636 (stale) Mirror 14: 0xbedbd636 (stale) Mirror 15: 0xbedbd636 (stale) Mirror 16: 0xbedbd636 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xbedbd636 (stale) Mirror 14: 0xbedbd636 (stale) Mirror 15: 0xbedbd636 (stale) Mirror 16: 0xbedbd636 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000688 1787575687.507205 punch from 0x0056b0 to 0x00d22a (0x7b7b bytes) 000689 1787575687.520289 mapwrite 0x00182c thru 0x006ed5 (0x56aa bytes) 000690 1787575687.612446 read 0x014dcc thru 0x01c1eb (0x7420 bytes) 000692 1787575687.638381 trunc from 0x040000 to 0x01a3a6 (0x25c59 bytes) 000693 1787575687.673630 trunc from 0x01a3a7 to 0x012d9e (0x7608 bytes) 000694 1787575687.689311 punch from 0x000880 to 0x0076a8 (0x6e29 bytes) 000695 1787575687.695234 trunc from 0x000b55 to 0x0025ee (0x1a9a bytes) 000696 1787575687.704438 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x12d9f): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12d9f): Mirror 9: 0xa9615538 (stale) Mirror 14: 0xa9615538 (stale) Mirror 15: 0xa9615538 (stale) Mirror 16: 0xa9615538 (stale) CRC-32 checksum value for chunk [0, 0x12d9f): Mirror 1: 0x2db41cff Mirror 17: 0x2db41cff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12d9f: skipped 000697 1787575688.708860 read 0x0125bd thru 0x012d9e (0x7e2 bytes) 000698 1787575688.727034 mapwrite 0x001457 thru 0x004db4 (0x395e bytes) 000699 1787575688.888110 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x12d9f): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12d9f): Mirror 9: 0xa9615538 (stale) Mirror 14: 0xa9615538 (stale) Mirror 15: 0xa9615538 (stale) Mirror 16: 0xa9615538 (stale) Mirror 17: 0x2db41cff (stale) CRC-32 checksum value for chunk [0, 0x12d9f): Mirror 1: 0x2d592cf6 Mirror 18: 0x2d592cf6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12d9f: skipped 000700 1787575690.179285 trunc from 0x012d9f to 0x00b608 (0x7796 bytes) 000701 1787575690.274994 fallocate 0x03a8d1 thru 0x03f015 (0x4745 bytes) 000703 1787575690.281758 read 0x00642e thru 0x00b608 (0x51db bytes) 000704 1787575690.291226 mapwrite 0x02d071 thru 0x036bfc (0x9b8c bytes) 000705 1787575690.360740 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x36bfd): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36bfd): Mirror 9: 0xe334a1d (stale) Mirror 14: 0xe334a1d (stale) Mirror 15: 0xe334a1d (stale) Mirror 16: 0xe334a1d (stale) Mirror 17: 0xceba5f1e (stale) Mirror 18: 0xfa256a0b (stale) CRC-32 checksum value for chunk [0, 0x36bfd): Mirror 1: 0x4a843ed7 Mirror 19: 0x4a843ed7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36bfd: skipped 000706 1787575691.870553 mapread 0x014401 thru 0x01f080 (0xac80 bytes) 000707 1787575691.935374 trunc from 0x0056cb to 0x00aaf2 (0x5428 bytes) 000708 1787575692.006464 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36bfd): Mirror 9: 0xe334a1d (stale) Mirror 14: 0xe334a1d (stale) Mirror 16: 0xe334a1d (stale) Mirror 17: 0xceba5f1e (stale) Mirror 18: 0xfa256a0b (stale) Mirror 19: 0x4a843ed7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36bfd): Mirror 9: 0xe334a1d (stale) Mirror 14: 0xe334a1d (stale) Mirror 16: 0xe334a1d (stale) Mirror 17: 0xceba5f1e (stale) Mirror 18: 0xfa256a0b (stale) Mirror 19: 0x4a843ed7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36bfd: skipped 000709 1787575692.545044 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36bfd): Mirror 9: 0xe334a1d (stale) Mirror 14: 0xe334a1d (stale) Mirror 16: 0xe334a1d (stale) Mirror 17: 0xceba5f1e (stale) Mirror 18: 0xfa256a0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36bfd): Mirror 9: 0xe334a1d (stale) Mirror 14: 0xe334a1d (stale) Mirror 16: 0xe334a1d (stale) Mirror 17: 0xceba5f1e (stale) Mirror 18: 0xfa256a0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36bfd: skipped 000710 1787575693.003575 read 0x01c0f3 thru 0x01e733 (0x2641 bytes) 000711 1787575693.011184 mapwrite 0x03a871 thru 0x03ddc4 (0x3554 bytes) 000712 1787575693.140180 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ddc5): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddc5): Mirror 9: 0xba49f303 (stale) Mirror 14: 0xba49f303 (stale) Mirror 16: 0xba49f303 (stale) Mirror 17: 0x99916163 (stale) Mirror 18: 0xec49a4bf (stale) CRC-32 checksum value for chunk [0, 0x3ddc5): Mirror 1: 0xaaea7f53 Mirror 19: 0xaaea7f53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddc5: skipped 000713 1787575694.642229 read 0x00b6f1 thru 0x00f2f7 (0x3c07 bytes) 000714 1787575694.671540 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ddc5): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ddc5): Mirror 9: 0xba49f303 (stale) Mirror 14: 0xba49f303 (stale) Mirror 16: 0xba49f303 (stale) Mirror 17: 0x99916163 (stale) Mirror 18: 0xec49a4bf (stale) CRC-32 checksum value for chunk [0, 0x3ddc5): Mirror 1: 0xaaea7f53 Mirror 19: 0xaaea7f53 Mirror 20: 0xaaea7f53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ddc5: skipped 000715 1787575695.885440 mapwrite 0x02aa45 thru 0x031e76 (0x7432 bytes) 000716 1787575695.998491 mapwrite 0x03b940 thru 0x03ffff (0x46c0 bytes) 000717 1787575696.100592 fallocate 0x014543 thru 0x021daa (0xd868 bytes) 000718 1787575696.107540 trunc from 0x040000 to 0x00eb77 (0x31488 bytes) 000719 1787575696.135393 trunc from 0x003469 to 0x00eb77 (0xb70f bytes) 000720 1787575696.143674 fallocate 0x0399a3 thru 0x03ffff (0x665d bytes) 000721 1787575696.167947 MIRROR_RESYNC 8 mirrors 000722 1787575697.706143 punch from 0x000ff0 to 0x00ccce (0xbcdf bytes) 000723 1787575697.790119 read 0x00e4a6 thru 0x01698a (0x84e5 bytes) 000724 1787575697.810118 read 0x0064b3 thru 0x0140d9 (0xdc27 bytes) 000725 1787575697.822926 mapread 0x026022 thru 0x02f003 (0x8fe2 bytes) 000726 1787575697.823995 punch from 0x02fd09 to 0x035ecb (0x61c3 bytes) 000727 1787575697.831286 trunc from 0x040000 to 0x0011f3 (0x3ee0c bytes) 000728 1787575697.861541 punch from 0x000fc7 to 0x0011f3 (0x22d bytes) 000729 1787575697.874971 trunc from 0x000fbc to 0x0011f3 (0x238 bytes) 000730 1787575697.880872 punch from 0x000d1a to 0x0011f3 (0x4da bytes) 000731 1787575697.892604 trunc from 0x0011f4 to 0x01ce02 (0x1bc0f bytes) 000732 1787575697.912705 punch from 0x005575 to 0x00db6e (0x85fa bytes) 000733 1787575697.918739 trunc from 0x01ce03 to 0x01a437 (0x29cb bytes) 000735 1787575697.971723 read 0x0070c5 thru 0x00a992 (0x38ce bytes) 000736 1787575697.998557 punch from 0x008d87 to 0x011995 (0x8c0f bytes) 000737 1787575698.006748 fallocate 0x01e259 thru 0x024237 (0x5fdf bytes) 000738 1787575698.024858 mapread 0x0025d6 thru 0x00385c (0x1287 bytes) 000739 1787575698.026156 fallocate 0x02da84 thru 0x0330d4 (0x5651 bytes) 000740 1787575698.032414 fallocate 0x0028db thru 0x00ceff (0xa625 bytes) 000741 1787575698.061699 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1a438): Mirror 9: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a438): Mirror 9: 0x51eaef94 (stale) Mirror 14: 0x51eaef94 (stale) Mirror 16: 0x51eaef94 (stale) Mirror 17: 0x51eaef94 (stale) Mirror 18: 0x51eaef94 (stale) Mirror 19: 0x51eaef94 (stale) Mirror 20: 0x51eaef94 (stale) CRC-32 checksum value for chunk [0, 0x1a438): Mirror 1: 0xb0b5bb27 Mirror 21: 0xb0b5bb27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a438: skipped 000742 1787575699.784115 mapwrite 0x0292de thru 0x035b84 (0xc8a7 bytes) 000743 1787575699.928694 read 0x00c7de thru 0x012b9a (0x63bd bytes) 000744 1787575699.944208 mapread 0x000437 thru 0x00708a (0x6c54 bytes) 000745 1787575699.953504 write 0x03f3f4 thru 0x03ffff (0xc0c bytes) 000746 1787575699.956115 write 0x023f29 thru 0x0242d4 (0x3ac bytes) 000748 1787575699.968736 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x3e134ae7 (stale) Mirror 16: 0x3e134ae7 (stale) Mirror 17: 0x3e134ae7 (stale) Mirror 18: 0x3e134ae7 (stale) Mirror 19: 0x3e134ae7 (stale) Mirror 20: 0x3e134ae7 (stale) Mirror 21: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x3e134ae7 (stale) Mirror 16: 0x3e134ae7 (stale) Mirror 17: 0x3e134ae7 (stale) Mirror 18: 0x3e134ae7 (stale) Mirror 19: 0x3e134ae7 (stale) Mirror 20: 0x3e134ae7 (stale) Mirror 21: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000749 1787575700.670748 read 0x033d08 thru 0x03ffff (0xc2f8 bytes) 000750 1787575700.680047 fallocate 0x021522 thru 0x025cbe (0x479d bytes) 000751 1787575700.684861 trunc from 0x040000 to 0x01c24b (0x23db4 bytes) 000752 1787575700.760071 mapwrite 0x0148b5 thru 0x01fe8f (0xb5db bytes) 000753 1787575700.900217 read 0x0054d6 thru 0x00dd39 (0x8864 bytes) 000754 1787575700.919829 punch from 0x001b1c to 0x00c291 (0xa776 bytes) 000755 1787575700.929354 trunc from 0x01fe90 to 0x01fa4f (0x440 bytes) 000756 1787575700.966425 trunc from 0x01fa50 to 0x00be3b (0x13c14 bytes) 000757 1787575700.996363 write 0x03707b thru 0x03ffff (0x8f85 bytes) 000759 1787575701.021083 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x3e134ae7 (stale) Mirror 16: 0x3e134ae7 (stale) Mirror 17: 0x3e134ae7 (stale) Mirror 18: 0x3e134ae7 (stale) Mirror 19: 0x3e134ae7 (stale) Mirror 20: 0x3e134ae7 (stale) Mirror 21: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd5992a5e Mirror 22: 0xd5992a5e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000760 1787575702.364870 trunc from 0x040000 to 0x001187 (0x3ee78 bytes) 000761 1787575702.446545 mapwrite 0x02fed9 thru 0x03ba98 (0xbbc0 bytes) 000762 1787575702.583872 fallocate 0x01d004 thru 0x02b2a8 (0xe2a5 bytes) 000763 1787575702.591287 mapwrite 0x010f5f thru 0x012293 (0x1335 bytes) 000764 1787575702.664013 mapread 0x00f7a8 thru 0x0100c1 (0x91a bytes) 000765 1787575702.671051 write 0x01b03e thru 0x01c52e (0x14f1 bytes) 000767 1787575702.692826 mapread 0x020369 thru 0x02e567 (0xe1ff bytes) 000768 1787575702.760299 fallocate 0x01987d thru 0x021f74 (0x86f8 bytes) 000769 1787575702.772320 read 0x013144 thru 0x01b01c (0x7ed9 bytes) 000770 1787575702.780066 mapread 0x019f80 thru 0x02898a (0xea0b bytes) 000771 1787575702.785183 fallocate 0x02e36e thru 0x03387c (0x550f bytes) 000772 1787575702.806206 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3ba99): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba99): Mirror 14: 0xf0303b7c (stale) Mirror 16: 0xf0303b7c (stale) Mirror 17: 0xf0303b7c (stale) Mirror 18: 0xf0303b7c (stale) Mirror 19: 0xf0303b7c (stale) Mirror 20: 0xf0303b7c (stale) Mirror 21: 0x7234d217 (stale) Mirror 22: 0x2112a822 (stale) CRC-32 checksum value for chunk [0, 0x3ba99): Mirror 1: 0xf1e6d6dd Mirror 23: 0xf1e6d6dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba99: skipped 000773 1787575704.280371 mapread 0x033c73 thru 0x03ba98 (0x7e26 bytes) 000774 1787575704.318143 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba99): Mirror 14: 0xf0303b7c (stale) Mirror 16: 0xf0303b7c (stale) Mirror 17: 0xf0303b7c (stale) Mirror 18: 0xf0303b7c (stale) Mirror 19: 0xf0303b7c (stale) Mirror 20: 0xf0303b7c (stale) Mirror 22: 0x2112a822 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba99): Mirror 14: 0xf0303b7c (stale) Mirror 16: 0xf0303b7c (stale) Mirror 17: 0xf0303b7c (stale) Mirror 18: 0xf0303b7c (stale) Mirror 19: 0xf0303b7c (stale) Mirror 20: 0xf0303b7c (stale) Mirror 22: 0x2112a822 (stale) CRC-32 checksum value for chunk [0, 0x3ba99): Mirror 1: 0xf1e6d6dd Mirror 23: 0xf1e6d6dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba99: skipped 000775 1787575705.978343 trunc from 0x03ba99 to 0x00be49 (0x2fc4f bytes) 000776 1787575706.068425 mapwrite 0x02bdf8 thru 0x02c90e (0xb17 bytes) 000777 1787575706.123089 punch from 0x009c3c to 0x00b061 (0x1426 bytes) 000779 1787575706.129487 write 0x02150d thru 0x022ec9 (0x19bd bytes) 000780 1787575706.140296 punch from 0x01c353 to 0x024846 (0x84f4 bytes) 000781 1787575706.148419 trunc from 0x02c90f to 0x02bf84 (0x98a bytes) 000782 1787575706.165954 punch from 0x00677d to 0x012671 (0xbef5 bytes) 000784 1787575706.171407 punch from 0x002224 to 0x00841c (0x61f9 bytes) 000786 1787575706.178043 mapwrite 0x015821 thru 0x020bf8 (0xb3d8 bytes) 000787 1787575706.239771 punch from 0x028182 to 0x02bf84 (0x3e03 bytes) 000789 1787575706.255427 MIRROR_RESYNC 9 mirrors 000790 1787575708.804717 MIRROR_RESYNC 9 mirrors 000791 1787575709.704669 mapwrite 0x01375a thru 0x01d71a (0x9fc1 bytes) 000792 1787575709.795937 MIRROR_RESYNC 9 mirrors 000793 1787575712.169952 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000794 1787575712.742360 punch from 0x019760 to 0x020e04 (0x76a5 bytes) 000795 1787575712.836498 fallocate 0x03f731 thru 0x03ffff (0x8cf bytes) 000796 1787575712.844526 mapread 0x002edc thru 0x011c67 (0xed8c bytes) 000797 1787575712.896695 write 0x016457 thru 0x026420 (0xffca bytes) 000798 1787575712.919544 punch from 0x029b10 to 0x030cdd (0x71ce bytes) 000799 1787575712.927328 fallocate 0x0268eb thru 0x02dd31 (0x7447 bytes) 000800 1787575712.938133 read 0x00e708 thru 0x0173f7 (0x8cf0 bytes) 000801 1787575712.938645 trunc from 0x040000 to 0x004268 (0x3bd97 bytes) 000802 1787575713.018011 trunc from 0x00104b to 0x004268 (0x321e bytes) 000803 1787575713.029372 mapwrite 0x0316fe thru 0x033166 (0x1a69 bytes) 000804 1787575713.093508 trunc from 0x033167 to 0x02215f (0x11007 bytes) 000805 1787575713.134361 mapwrite 0x028e01 thru 0x038703 (0xf903 bytes) 000806 1787575713.238337 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38704): Mirror 14: 0xb10a5343 (stale) Mirror 16: 0xb10a5343 (stale) Mirror 17: 0xb10a5343 (stale) Mirror 18: 0xb10a5343 (stale) Mirror 19: 0xb10a5343 (stale) Mirror 22: 0xb10a5343 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38704): Mirror 14: 0xb10a5343 (stale) Mirror 16: 0xb10a5343 (stale) Mirror 17: 0xb10a5343 (stale) Mirror 18: 0xb10a5343 (stale) Mirror 19: 0xb10a5343 (stale) Mirror 22: 0xb10a5343 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38704: skipped 000807 1787575713.789645 fallocate 0x034502 thru 0x03da10 (0x950f bytes) 000808 1787575713.798517 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3da11): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da11): Mirror 14: 0x631f68d0 (stale) Mirror 16: 0x631f68d0 (stale) Mirror 17: 0x631f68d0 (stale) Mirror 18: 0x631f68d0 (stale) Mirror 19: 0x631f68d0 (stale) Mirror 22: 0x631f68d0 (stale) CRC-32 checksum value for chunk [0, 0x3da11): Mirror 1: 0x71a692ac Mirror 23: 0x71a692ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da11: skipped 000809 1787575715.019567 trunc from 0x0371a2 to 0x03da10 (0x686f bytes) 000811 1787575715.097277 mapread 0x009cce thru 0x00f2e3 (0x5616 bytes) 000812 1787575715.148480 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3da11): Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da11): Mirror 14: 0x631f68d0 (stale) Mirror 16: 0x631f68d0 (stale) Mirror 17: 0x631f68d0 (stale) Mirror 18: 0x631f68d0 (stale) Mirror 19: 0x631f68d0 (stale) Mirror 22: 0x631f68d0 (stale) Mirror 23: 0x71a692ac (stale) CRC-32 checksum value for chunk [0, 0x3da11): Mirror 1: 0x5d93244a Mirror 24: 0x5d93244a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da11: skipped 000813 1787575716.325050 trunc from 0x03da11 to 0x016896 (0x2717a bytes) 000814 1787575716.377424 fallocate 0x0171e1 thru 0x021a8a (0xa8aa bytes) 000815 1787575716.382617 punch from 0x001a06 to 0x00aab4 (0x90af bytes) 000816 1787575716.386847 read 0x013f58 thru 0x01f7ca (0xb873 bytes) 000817 1787575716.396282 trunc from 0x021a8b to 0x0260c2 (0x4638 bytes) 000818 1787575716.414270 fallocate 0x037b9b thru 0x03f36e (0x77d4 bytes) 000819 1787575716.419235 fallocate 0x035404 thru 0x0370d6 (0x1cd3 bytes) 000820 1787575716.423892 write 0x004689 thru 0x010892 (0xc20a bytes) 000821 1787575716.433748 punch from 0x00738d to 0x009428 (0x209c bytes) 000822 1787575716.446771 fallocate 0x03b42e thru 0x03d3a3 (0x1f76 bytes) 000823 1787575716.452323 trunc from 0x02a9ce to 0x02f207 (0x483a bytes) 000825 1787575716.457770 write 0x0040db thru 0x0086d1 (0x45f7 bytes) 000826 1787575716.466683 MIRROR_RESYNC 9 mirrors 000827 1787575718.878915 mapwrite 0x038689 thru 0x03ba72 (0x33ea bytes) 000828 1787575719.041682 trunc from 0x0368e8 to 0x03e07f (0x7798 bytes) 000829 1787575719.056956 fallocate 0x0116e4 thru 0x017ddf (0x66fc bytes) 000830 1787575719.082259 trunc from 0x03f36f to 0x01d747 (0x21c27 bytes) 000831 1787575719.135566 MIRROR_RESYNC 9 mirrors 000832 1787575720.792090 trunc from 0x00f6f5 to 0x01cdde (0xd6ea bytes) 000833 1787575720.839612 read 0x000f17 thru 0x00c1ae (0xb298 bytes) 000834 1787575720.862196 fallocate 0x03a09f thru 0x03ffff (0x5f61 bytes) 000835 1787575720.874093 fallocate 0x01dae4 thru 0x02d9e2 (0xfeff bytes) 000836 1787575720.885314 punch from 0x00dc04 to 0x018cad (0xb0aa bytes) 000837 1787575720.892678 fallocate 0x03f35d thru 0x03ff6f (0xc13 bytes) 000838 1787575720.899231 trunc from 0x011c32 to 0x01c071 (0xa440 bytes) 000839 1787575720.905689 mapread 0x024f0f thru 0x029bfc (0x4cee bytes) 000840 1787575720.952551 punch from 0x02621e to 0x029560 (0x3343 bytes) 000841 1787575720.965816 mapwrite 0x02b371 thru 0x02fd13 (0x49a3 bytes) 000842 1787575721.080278 read 0x000a00 thru 0x006a3f (0x6040 bytes) 000843 1787575721.080908 trunc from 0x02d67f to 0x02fd13 (0x2695 bytes) 000844 1787575721.087519 trunc from 0x009ddd to 0x01438f (0xa5b3 bytes) 000845 1787575721.093747 punch from 0x02b04c to 0x02fd13 (0x4cc8 bytes) 000846 1787575721.113419 fallocate 0x0316c7 thru 0x039c8c (0x85c6 bytes) 000848 1787575721.121653 write 0x030f5d thru 0x03bce9 (0xad8d bytes) 000849 1787575721.132079 mapwrite 0x007143 thru 0x00b8c4 (0x4782 bytes) 000850 1787575721.147474 read 0x002d6d thru 0x00fbbb (0xce4f bytes) 000851 1787575721.148088 trunc from 0x01926e to 0x01b3e5 (0x2178 bytes) 000852 1787575721.151208 punch from 0x0300f3 to 0x0340eb (0x3ff9 bytes) 000853 1787575721.157104 mapwrite 0x022130 thru 0x0243b6 (0x2287 bytes) 000854 1787575721.176088 read 0x0090b1 thru 0x00c146 (0x3096 bytes) 000855 1787575721.177494 punch from 0x026f72 to 0x029495 (0x2524 bytes) 000856 1787575721.185374 mapwrite 0x03c7d5 thru 0x03ffff (0x382b bytes) 000857 1787575721.235270 write 0x0283c8 thru 0x034b8b (0xc7c4 bytes) 000858 1787575721.244383 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x16a301dc (stale) Mirror 16: 0x16a301dc (stale) Mirror 17: 0x16a301dc (stale) Mirror 18: 0x16a301dc (stale) Mirror 19: 0x16a301dc (stale) Mirror 22: 0x16a301dc (stale) Mirror 24: 0x16a301dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x16a301dc (stale) Mirror 16: 0x16a301dc (stale) Mirror 17: 0x16a301dc (stale) Mirror 18: 0x16a301dc (stale) Mirror 19: 0x16a301dc (stale) Mirror 22: 0x16a301dc (stale) Mirror 24: 0x16a301dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000859 1787575721.693765 write 0x02925b thru 0x0382b2 (0xf058 bytes) 000860 1787575721.732356 mapread 0x01eeb0 thru 0x0242ec (0x543d bytes) 000866 1787575721.771993 write 0x004437 thru 0x00e963 (0xa52d bytes) 000867 1787575721.792692 read 0x022910 thru 0x029c0d (0x72fe bytes) 000868 1787575721.795575 mapread 0x028df9 thru 0x02ce34 (0x403c bytes) 000869 1787575721.796474 trunc from 0x040000 to 0x0155f2 (0x2aa0d bytes) 000870 1787575721.831042 fallocate 0x03869c thru 0x03d293 (0x4bf8 bytes) 000871 1787575721.845329 MIRROR_RESYNC 8 mirrors 000873 1787575723.537623 MIRROR_RESYNC 8 mirrors 000874 1787575724.277651 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) 000875 1787575724.627064 write 0x01c524 thru 0x02be51 (0xf92e bytes) 000876 1787575724.681685 mapwrite 0x029d70 thru 0x037b3e (0xddcf bytes) 000877 1787575724.749178 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37b3f): Mirror 16: 0x9e77ad22 (stale) Mirror 17: 0x9e77ad22 (stale) Mirror 18: 0x9e77ad22 (stale) Mirror 22: 0x9e77ad22 (stale) Mirror 24: 0x9e77ad22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b3f): Mirror 16: 0x9e77ad22 (stale) Mirror 17: 0x9e77ad22 (stale) Mirror 18: 0x9e77ad22 (stale) Mirror 22: 0x9e77ad22 (stale) Mirror 24: 0x9e77ad22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b3f: skipped 000878 1787575725.324714 read 0x020012 thru 0x02effd (0xefec bytes) 000879 1787575725.335925 trunc from 0x037b3f to 0x00135f (0x367df bytes) 000880 1787575725.358738 mapread 0x0006dc thru 0x00135f (0xc84 bytes) 000881 1787575725.371937 fallocate 0x006b7b thru 0x00cc4b (0x60d1 bytes) 000882 1787575725.377525 mapread 0x006852 thru 0x00cc4b (0x63fa bytes) 000883 1787575725.394988 read 0x001b13 thru 0x0074c3 (0x59b1 bytes) 000884 1787575725.395607 mapread 0x00b436 thru 0x00cc4b (0x1816 bytes) 000885 1787575725.396426 read 0x0040da thru 0x00cc4b (0x8b72 bytes) 000886 1787575725.398669 punch from 0x000c5f to 0x002fff (0x23a1 bytes) 000887 1787575725.404622 trunc from 0x00cc4c to 0x037f89 (0x2b33e bytes) 000889 1787575725.432125 trunc from 0x01f840 to 0x0233cd (0x3b8e bytes) 000891 1787575725.444155 trunc from 0x004f2c to 0x006961 (0x1a36 bytes) 000892 1787575725.454620 punch from 0x023929 to 0x0275d4 (0x3cac bytes) 000893 1787575725.476949 punch from 0x00d766 to 0x00d7c0 (0x5b bytes) 000894 1787575725.481193 read 0x00a91b thru 0x010579 (0x5c5f bytes) 000895 1787575725.526012 trunc from 0x037f8a to 0x00fdea (0x2819f bytes) 000896 1787575725.544193 mapwrite 0x031c03 thru 0x036101 (0x44ff bytes) 000897 1787575725.604974 MIRROR_RESYNC 6 mirrors 000898 1787575726.899427 read 0x016a27 thru 0x022c85 (0xc25f bytes) 000899 1787575726.912602 write 0x03ee5e thru 0x03fc22 (0xdc5 bytes) 000900 1787575726.940502 mapread 0x021d27 thru 0x022266 (0x540 bytes) 000901 1787575726.954256 write 0x011c0f thru 0x01f034 (0xd426 bytes) 000902 1787575726.966085 punch from 0x003b41 to 0x00a660 (0x6b20 bytes) 000903 1787575726.970995 trunc from 0x03fc23 to 0x0001a3 (0x3fa7f bytes) 000904 1787575727.001111 MIRROR_RESYNC 6 mirrors 000905 1787575727.758098 mapread 0x000151 thru 0x0001a3 (0x53 bytes) 000906 1787575727.762700 mapwrite 0x00ef8e thru 0x013c5d (0x4cd0 bytes) 000907 1787575727.838618 punch from 0x002fc3 to 0x00a2a5 (0x72e3 bytes) 000908 1787575727.844089 fallocate 0x013361 thru 0x018aff (0x579f bytes) 000909 1787575727.850005 read 0x012bc8 thru 0x018aff (0x5f38 bytes) 000911 1787575727.860148 trunc from 0x0189bb to 0x018aff (0x145 bytes) 000912 1787575727.864925 fallocate 0x038f09 thru 0x03ffff (0x70f7 bytes) 000913 1787575727.872904 trunc from 0x018b00 to 0x00f54a (0x95b5 bytes) 000914 1787575727.895687 read 0x00ad12 thru 0x00f54a (0x4839 bytes) 000915 1787575727.915451 mapread 0x006546 thru 0x00f54a (0x9005 bytes) 000916 1787575727.925840 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xf54b): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf54b): Mirror 16: 0x75dc3f1a (stale) Mirror 17: 0x75dc3f1a (stale) Mirror 18: 0x75dc3f1a (stale) Mirror 22: 0x75dc3f1a (stale) Mirror 24: 0x75dc3f1a (stale) CRC-32 checksum value for chunk [0, 0xf54b): Mirror 1: 0x8e063ff8 Mirror 25: 0x8e063ff8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf54b: skipped 000917 1787575728.863418 punch from 0x007819 to 0x00f54a (0x7d32 bytes) 000918 1787575728.918412 read 0x00942c thru 0x00be3c (0x2a11 bytes) 000919 1787575728.926278 trunc from 0x000467 to 0x00bf8a (0xbb24 bytes) 000920 1787575728.931389 read 0x00481f thru 0x006b5d (0x233f bytes) 000921 1787575728.957218 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf54b): Mirror 16: 0x75dc3f1a (stale) Mirror 17: 0x75dc3f1a (stale) Mirror 22: 0x75dc3f1a (stale) Mirror 24: 0x75dc3f1a (stale) Mirror 25: 0x8e063ff8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf54b): Mirror 16: 0x75dc3f1a (stale) Mirror 17: 0x75dc3f1a (stale) Mirror 22: 0x75dc3f1a (stale) Mirror 24: 0x75dc3f1a (stale) Mirror 25: 0x8e063ff8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf54b: skipped 000922 1787575729.376712 mapread 0x005712 thru 0x00f54a (0x9e39 bytes) 000923 1787575729.401544 mapread 0x006e82 thru 0x00f54a (0x86c9 bytes) 000924 1787575729.402764 punch from 0x009461 to 0x00b642 (0x21e2 bytes) 000925 1787575729.407924 write 0x02723f thru 0x02edb5 (0x7b77 bytes) 000926 1787575729.416210 trunc from 0x02edb6 to 0x03dfc7 (0xf212 bytes) 000927 1787575729.434071 mapread 0x03aec1 thru 0x03bddc (0xf1c bytes) 000928 1787575729.449474 read 0x03499d thru 0x0389f8 (0x405c bytes) 000930 1787575729.451431 trunc from 0x002f44 to 0x004bf7 (0x1cb4 bytes) 000931 1787575729.456485 punch from 0x0386cb to 0x03dfc7 (0x58fd bytes) 000932 1787575729.461949 mapwrite 0x03f499 thru 0x03ffff (0xb67 bytes) 000933 1787575729.507124 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xe20eea22 (stale) Mirror 17: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 25: 0x533c229d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf33c11c1 Mirror 26: 0xf33c11c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000934 1787575730.759487 mapread 0x03d132 thru 0x03ffff (0x2ece bytes) 000936 1787575730.812835 trunc from 0x020a20 to 0x02d310 (0xc8f1 bytes) 000937 1787575730.867223 mapwrite 0x00d069 thru 0x0199c4 (0xc95c bytes) 000938 1787575730.906138 fallocate 0x02021c thru 0x02bc80 (0xba65 bytes) 000939 1787575730.911077 trunc from 0x0042d5 to 0x007f9b (0x3cc7 bytes) 000940 1787575730.915458 trunc from 0x006ee2 to 0x014222 (0xd341 bytes) 000941 1787575730.920736 trunc from 0x026d46 to 0x032f54 (0xc20f bytes) 000942 1787575730.925824 punch from 0x01f5c7 to 0x020c20 (0x165a bytes) 000943 1787575730.934097 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 25: 0x533c229d (stale) Mirror 26: 0xf33c11c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 25: 0x533c229d (stale) Mirror 26: 0xf33c11c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000944 1787575731.269112 mapwrite 0x0027f2 thru 0x010dfc (0xe60b bytes) 000945 1787575731.304413 punch from 0x00b857 to 0x00c6d4 (0xe7e bytes) 000946 1787575731.309492 mapwrite 0x00f796 thru 0x014a32 (0x529d bytes) 000947 1787575731.339173 read 0x00119e thru 0x006cec (0x5b4f bytes) 000948 1787575731.344246 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xe20eea22 (stale) Mirror 22: 0xe20eea22 (stale) Mirror 24: 0xe20eea22 (stale) Mirror 25: 0x533c229d (stale) Mirror 26: 0xf33c11c1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa176f162 Mirror 27: 0xa176f162 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000950 1787575732.519360 trunc from 0x040000 to 0x03732b (0x8cd4 bytes) 000952 1787575732.618407 read 0x0188d3 thru 0x0209b1 (0x80df bytes) 000953 1787575732.631331 mapread 0x021da7 thru 0x02888e (0x6ae8 bytes) 000954 1787575732.650262 mapread 0x028236 thru 0x029768 (0x1533 bytes) 000955 1787575732.651616 trunc from 0x02b294 to 0x03596a (0xa6d7 bytes) 000956 1787575732.657741 mapread 0x008543 thru 0x012532 (0x9ff0 bytes) 000957 1787575732.695009 MIRROR_RESYNC 7 mirrors 000958 1787575734.244136 trunc from 0x03732c to 0x02b690 (0xbc9b bytes) 000959 1787575734.287368 fallocate 0x01b1e2 thru 0x028ae3 (0xd902 bytes) 000962 1787575734.293181 trunc from 0x02b691 to 0x03dcdf (0x1264f bytes) 000963 1787575734.317429 read 0x02f6ac thru 0x034958 (0x52ad bytes) 000964 1787575734.329907 mapread 0x03571a thru 0x03dcdf (0x85c6 bytes) 000965 1787575734.349948 write 0x02d191 thru 0x02d6ba (0x52a bytes) 000966 1787575734.355119 mapread 0x016c26 thru 0x01e2be (0x7699 bytes) 000967 1787575734.370466 read 0x0309f4 thru 0x03901e (0x862b bytes) 000968 1787575734.371459 trunc from 0x01763f to 0x01e372 (0x6d34 bytes) 000969 1787575734.376910 trunc from 0x03dce0 to 0x00d9c1 (0x3031e bytes) 000970 1787575734.403074 mapwrite 0x001af6 thru 0x00718a (0x5695 bytes) 000971 1787575734.446322 write 0x00a0aa thru 0x0170a7 (0xcffe bytes) 000972 1787575734.453310 write 0x02b46e thru 0x02ebe6 (0x3779 bytes) 000973 1787575734.454589 trunc from 0x02ebe7 to 0x0111dc (0x1da0a bytes) 000974 1787575734.474611 read 0x00747f thru 0x0111dc (0x9d5e bytes) 000975 1787575734.475269 mapwrite 0x03607b thru 0x03ffff (0x9f85 bytes) 000977 1787575734.532425 mapread 0x0376e0 thru 0x03ffff (0x8920 bytes) 000978 1787575734.533520 punch from 0x0211af to 0x027971 (0x67c3 bytes) 000979 1787575734.539707 mapwrite 0x00e517 thru 0x01c687 (0xe171 bytes) 000980 1787575734.564919 fallocate 0x011821 thru 0x013f6e (0x274e bytes) 000981 1787575734.569633 punch from 0x03026e to 0x03061b (0x3ae bytes) 000982 1787575734.587705 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x1c1daa83 (stale) Mirror 22: 0x1c1daa83 (stale) Mirror 24: 0x1c1daa83 (stale) Mirror 25: 0x1c1daa83 (stale) Mirror 26: 0x1c1daa83 (stale) Mirror 27: 0x1c1daa83 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4ca92e0d Mirror 28: 0x4ca92e0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000983 1787575735.320223 write 0x01316d thru 0x01c31b (0x91af bytes) 000984 1787575735.391151 trunc from 0x02f6d5 to 0x0377fd (0x8129 bytes) 000985 1787575735.396437 mapread 0x0207df thru 0x02f240 (0xea62 bytes) 000986 1787575735.433178 write 0x028915 thru 0x02b35d (0x2a49 bytes) 000988 1787575735.441550 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x1c1daa83 (stale) Mirror 24: 0x1c1daa83 (stale) Mirror 25: 0x1c1daa83 (stale) Mirror 26: 0x1c1daa83 (stale) Mirror 27: 0x1c1daa83 (stale) Mirror 28: 0x4ca92e0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x1c1daa83 (stale) Mirror 24: 0x1c1daa83 (stale) Mirror 25: 0x1c1daa83 (stale) Mirror 26: 0x1c1daa83 (stale) Mirror 27: 0x1c1daa83 (stale) Mirror 28: 0x4ca92e0d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000989 1787575735.919642 mapread 0x00a178 thru 0x0187f1 (0xe67a bytes) 000990 1787575735.945031 trunc from 0x040000 to 0x005185 (0x3ae7a bytes) 000991 1787575735.990906 read 0x003d02 thru 0x005185 (0x1484 bytes) 000992 1787575736.005746 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x5186): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5186): Mirror 22: 0x21787c29 (stale) Mirror 24: 0x21787c29 (stale) Mirror 25: 0x21787c29 (stale) Mirror 26: 0x21787c29 (stale) Mirror 27: 0x21787c29 (stale) Mirror 28: 0x9e7567f (stale) CRC-32 checksum value for chunk [0, 0x5186): Mirror 1: 0x9e7567f Mirror 29: 0x9e7567f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5186: skipped 000993 1787575736.879666 punch from 0x000c7c to 0x005185 (0x450a bytes) 000994 1787575736.940379 mapread 0x000150 thru 0x0034aa (0x335b bytes) 000995 1787575736.965163 write 0x02aecf thru 0x02c882 (0x19b4 bytes) 000996 1787575736.971391 trunc from 0x01c836 to 0x0261ba (0x9985 bytes) 000997 1787575736.977207 mapwrite 0x00cce0 thru 0x015499 (0x87ba bytes) 000998 1787575737.022290 punch from 0x00599e to 0x005a31 (0x94 bytes) 000999 1787575737.027282 trunc from 0x02c883 to 0x02c419 (0x469 bytes) All operations completed A-OK! PASS 70a (137s) == sanity-flr test 71: check mirror extend parent fid ==== 08:49:02 (1787575742) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 08:49:14 (1787575754) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.291591 s, 7.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131934 s, 7.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fe:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c2:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ff:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 08:49:24 (1787575764) Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6943264 .. Extending file size to 1477280 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7420256 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8227680 .. Extending file size to 7736480 .. Extending file size to 8282016 .. done Extending file size to 8380064 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8477280 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9005664 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61332 61333 61334 61336 61337 done 192.168.204.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg459-client.virtnet /mnt/lustre2 (opts:) 192.168.204.159@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg459-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:50:42 (1787575842) Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6875264 .. Extending file size to 6929440 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8532608 .. Extending file size to 9049856 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8866816 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62910 62911 62913 62915 62917 done 192.168.204.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg459-client.virtnet /mnt/lustre2 (opts:) 192.168.204.159@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg459-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:52:07 (1787575927) Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2652 1268652 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25148 3581872 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17496 3589524 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42644 7171396 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2652 1268652 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1860 1269444 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25148 3581872 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17496 3589524 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42644 7171396 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.171277 s, 12.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.217998 s, 9.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xdbcc4ac4 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg459-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:52:25 (1787575945) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123779 s, 16.9 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 08:52:31 (1787575951) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e2:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d8:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d8:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d9:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x314:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:52:38 (1787575958) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0794072 s, 26.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:52:45 (1787575965) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0949197 s, 22.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2db:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x315:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x316:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:52:53 (1787575973) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192621 s, 10.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x317:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x317:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x318:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:53:27 (1787576007) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg459-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770797 s, 13.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:54:22 (1787576062) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg459-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138002 s, 15.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2dd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:55:19 (1787576119) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg459-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg459-server: Pool lustre.test_204f created Adding targets to pool oleg459-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg459-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145599 s, 14.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg459-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg459-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg459-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:56:34 (1787576194) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:56:42 (1787576202) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:56:49 (1787576209) oleg459-server: Pool lustre.test_206 created Adding targets to pool oleg459-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg459-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg459-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg459-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg459-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:57:18 (1787576238) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617567 s, 17.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412304 s, 25.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9825822e Mirror 3: 0x9825822e PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:57:28 (1787576248) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:57:32 (1787576252) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:57:36 (1787576256) Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.083029 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1980000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1700000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1700000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1700000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1100000 Read this (4 bytes): BBBB 192.168.204.159@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg459-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:58:13 (1787576293) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.046343 s, 177 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:58:20 (1787576300) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476265 s, 22.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x203 lmm_fid: [0x200000405:0x203:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x203 lmm_fid: [0x200000405:0x203:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611651 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 08:58:22.000000000 -0400 Modify: 2026-08-24 08:58:22.000000000 -0400 Change: 2026-08-24 08:58:23.000000000 -0400 Birth: 2026-08-24 08:58:21.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:58:28 (1787576308) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412385 s, 25.4 MB/s oleg459-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg459-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg459-server: O/280000401/d5/325: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:58:49 (1787576329) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.247916 s, 165 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0266655 s, 154 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 08:58 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x208:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 08:58 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:58:56 (1787576336) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:59:05 (1787576345) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1879 sec =========== 08:59:15 (1787576355) === sanity-flr: start cleanup 08:59:17 (1787576357) === === sanity-flr: finish cleanup 08:59:26 (1787576366) ===