-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 09:54:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg121-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 09:54:27 (1787579667) === oleg121-client.virtnet: executing check_config_client /mnt/lustre oleg121-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg121-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff992ad0f0f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff992ad0f0f800.idle_timeout=debug disable quota as required oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 09:55:11 (1787579711) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8244] == sanity-flr test 0a: lfs mirror create with -N option == 09:55:14 (1787579714) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:55:31 (1787579731) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (5s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:55:36 (1787579736) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (6s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:55:42 (1787579742) PASS 0d (33s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:56:15 (1787579775) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:56:19 (1787579779) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 09:56:23 (1787579783) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff992ad0f0f800.stats=clear osc.lustre-OST0001-osc-ffff992ad0f0f800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:56:30 (1787579790) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:56:40 (1787579800) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:56:48 (1787579808) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:56:52 (1787579812) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:57:00 (1787579820) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 09:57:08 (1787579828) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:57:14 (1787579834) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:57:20 (1787579840) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:57:26 (1787579846) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:57:35 (1787579855) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:57:41 (1787579861) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735513 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536799 s, 19.5 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:57:48 (1787579868) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.324942 s, 32.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:57:57 (1787579877) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:58:09 (1787579889) Stopping /mnt/lustre-ost1 (opts:) on oleg121-server oleg121-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff992ad0f0f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff992ad0f0f800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg121-server: oleg121-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg121-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg121-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg121-server oleg121-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff992ad0f0f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff992ad0f0f800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg121-server: oleg121-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg121-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (134s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:00:24 (1787580024) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.474542 s, 15.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg121-server oleg121-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff992ad0f0f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff992ad0f0f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg121-server: oleg121-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg121-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:01:07 (1787580067) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg121-server oleg121-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff992ad0f0f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff992ad0f0f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg121-server: oleg121-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg121-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg121-server oleg121-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff992ad0f0f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff992ad0f0f800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg121-server: oleg121-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg121-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:02:21 (1787580141) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.88984 s, 18.1 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 10:02 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.875s user 0m0.001s sys 0m0.171s Stopping /mnt/lustre-ost1 (opts:) on oleg121-server oleg121-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff992ad0f0f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff992ad0f0f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.035s user 0m0.002s sys 0m0.180s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg121-server: oleg121-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg121-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg121-server oleg121-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff992ad0f0f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff992ad0f0f800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.804s user 0m0.000s sys 0m0.168s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg121-server: oleg121-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg121-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg121-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff992ad0f0f800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff992ad0f0f800.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:03:56 (1787580236) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:04:00 (1787580240) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:04:03 (1787580243) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 10:04:07 (1787580247) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 10:04:14 (1787580254) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.533648 s, 23.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:04:28 (1787580268) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05696 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230826 s, 45.4 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:04:36 (1787580276) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:04:46 (1787580286) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000796804 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000899776 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00132432 s, 773 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00244318 s, 419 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 10:04:56 (1787580296) create test files with size 15745359 .. 7a10dd87c251f68489e4e068b8c94685 - 41110fd96fb9f04ec027d2f755f89869 - 13ae2d4436b58cc14bc2261b402ca048 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg121-client.virtnet /mnt/lustre (opts:) Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 10:05:17 (1787580317) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:05:31 (1787580331) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:05:38 (1787580338) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03442 s, 30.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457413 s, 22.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 10:05:47 (1787580347) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0724044 s, 29.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.305689 s, 13.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146131 s, 14.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 10:06:02 (1787580362) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 10:06:05 (1787580365) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:06:08 (1787580368) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041423 s, 25.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 10:06:17 (1787580377) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.196718 s, 16.0 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:06:26 (1787580386) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:06:36 (1787580396) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.2574 s, 40.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg121-client.virtnet /mnt/lustre (opts:) Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:06:48 (1787580408) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff992ad0f0a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.407798 s, 25.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg121-client.virtnet /mnt/lustre (opts:) Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff992ad27d4000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:07:04 (1787580424) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:07:42 (1787580462) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:08:23 (1787580503) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 10:08:27 (1787580507) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.221271 s, 14.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.221105 s, 14.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x39053293 (nosync,stale) Mirror 2: 0x39053293 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 0b2bf3fb63764684483c2c13ae1c83fc - mirror 1 checksum: 2ed2e4a34bfa383f17ebe28d636b451b - mirror 2 checksum: 0b2bf3fb63764684483c2c13ae1c83fc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T10:08:30-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T10:08:30-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 0b2bf3fb63764684483c2c13ae1c83fc - mirror 1 checksum: 2ed2e4a34bfa383f17ebe28d636b451b - mirror 2 checksum: 2ed2e4a34bfa383f17ebe28d636b451b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T10:08:30-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T10:08:30-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:08:39 (1787580519) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032682 s, 32.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:08:48 (1787580528) Failing mds1 on oleg121-server Stopping /mnt/lustre-mds1 (opts:) on oleg121-server 10:09:00 (1787580540) shut down facet: mds1 facet_host: oleg121-server facet_failover_host: oleg121-server Failover mds1 to oleg121-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0000 10:09:31 (1787580571) targets are mounted 10:09:31 (1787580571) facet_failover done oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502119 s, 20.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (60s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:09:49 (1787580589) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0446252 s, 459 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0144594 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:09:58 (1787580598) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0841424 s, 243 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00690128 s, 3.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:10:07 (1787580607) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0171709 s, 954 kB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:10:15 (1787580615) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0470638 s, 435 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:10:23 (1787580623) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00502754 s, 815 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0287127 s, 143 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:10:31 (1787580631) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xf:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:10:51 (1787580651) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:11:19 (1787580679) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:11:46 (1787580706) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:12:12 (1787580732) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 10:12:20 (1787580740) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57614 mirror split pid 57615 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57614 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57615 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 10:13:26 (1787580806) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 808 000002 1787580808.462528 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x37df3 000003 1787580808.866468 fallocate 0x03791c thru 0x037df2 (0x4d7 bytes) 000004 1787580808.901798 mapwrite 0x0319bc thru 0x03e760 (0xcda5 bytes) 000005 1787580809.005538 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000006 1787580809.145122 fallocate 0x019d0a thru 0x01a4d8 (0x7cf bytes) 000007 1787580809.154898 read 0x013b22 thru 0x01bea1 (0x8380 bytes) 000009 1787580809.164739 mapread 0x0137d6 thru 0x01c8ce (0x90f9 bytes) 000010 1787580809.169912 fallocate 0x02d490 thru 0x02d6e3 (0x254 bytes) 000011 1787580809.175801 trunc from 0x0116ed to 0x01ae99 (0x97ad bytes) 000012 1787580809.179884 mapwrite 0x03a36a thru 0x03b2e5 (0xf7c bytes) 000013 1787580809.200100 fallocate 0x00db3f thru 0x01622c (0x86ee bytes) 000014 1787580809.204057 mapwrite 0x0329b3 thru 0x037abe (0x510c bytes) 000015 1787580809.227995 mapread 0x01071d thru 0x02046a (0xfd4e bytes) 000017 1787580809.243633 mapread 0x006570 thru 0x00f7d9 (0x926a bytes) 000018 1787580809.255412 MIRROR_EXTEND to 2 mirrors 000019 1787580809.682989 trunc from 0x03e761 to 0x02e7ae (0xffb2 bytes) 000020 1787580809.745472 trunc from 0x0253d1 to 0x02e7ae (0x93de bytes) 000021 1787580809.750002 trunc from 0x02e7af to 0x02b229 (0x3585 bytes) 000022 1787580809.768057 punch from 0x019e27 to 0x0211ad (0x7387 bytes) 000023 1787580809.773200 mapread 0x005b3b thru 0x0080b0 (0x2576 bytes) 000024 1787580809.790754 trunc from 0x02b22a to 0x01ce12 (0xe417 bytes) truncating to largest ever: 0x3bc4e 000025 1787580809.804476 trunc from 0x01ce13 to 0x03bc4d (0x1ee3b bytes) 000026 1787580809.818563 read 0x03b41e thru 0x03bc4d (0x830 bytes) 000028 1787580809.824959 punch from 0x0248b4 to 0x03000f (0xb75c bytes) 000029 1787580809.829999 mapread 0x0336b9 thru 0x03bc4d (0x8595 bytes) 000030 1787580809.837637 trunc from 0x016be7 to 0x0172d7 (0x6f1 bytes) 000031 1787580809.842901 fallocate 0x01efbe thru 0x024d71 (0x5db4 bytes) 000034 1787580809.848114 write 0x03d860 thru 0x03ffff (0x27a0 bytes) 000035 1787580809.852929 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb39de5bc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2f7b6fae Mirror 3: 0x2f7b6fae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000036 1787580810.526171 trunc from 0x040000 to 0x018e77 (0x27188 bytes) 000038 1787580810.575880 write 0x00c7fc thru 0x010cd7 (0x44dc bytes) fallocating to largest ever: 0x3dd08 000039 1787580810.586428 fallocate 0x03520f thru 0x03dd07 (0x8af9 bytes) 000040 1787580810.591215 read 0x0039a2 thru 0x009bdf (0x623e bytes) 000041 1787580810.603748 read 0x010aed thru 0x014e7e (0x4392 bytes) 000043 1787580810.621036 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dd08): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd08): Mirror 2: 0xc696dcd3 (stale) Mirror 3: 0x8180185b (stale) CRC-32 checksum value for chunk [0, 0x3dd08): Mirror 1: 0x9075181 Mirror 4: 0x9075181 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd08: skipped 000045 1787580811.314551 write 0x0308b1 thru 0x03a983 (0xa0d3 bytes) 000046 1787580811.362953 trunc from 0x03dd08 to 0x011c41 (0x2c0c6 bytes) 000047 1787580811.407060 mapread 0x010260 thru 0x011c41 (0x19e2 bytes) 000048 1787580811.420651 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11c42): Mirror 2: 0xa10daf86 (stale) Mirror 4: 0x36fd1706 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11c42): Mirror 2: 0xa10daf86 (stale) Mirror 4: 0x36fd1706 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11c42: skipped 000049 1787580811.660715 mapwrite 0x025b18 thru 0x026196 (0x67f bytes) 000050 1787580811.702074 write 0x038fc2 thru 0x03ffff (0x703e bytes) 000051 1787580811.708850 MIRROR_RESYNC 3 mirrors 000052 1787580812.430243 write 0x02b220 thru 0x030619 (0x53fa bytes) 000053 1787580812.461634 trunc from 0x040000 to 0x00d565 (0x32a9a bytes) 000054 1787580812.476790 write 0x035ed6 thru 0x03ffff (0xa12a bytes) 000055 1787580812.478534 punch from 0x0202fe to 0x02d432 (0xd135 bytes) 000056 1787580812.482128 mapwrite 0x016201 thru 0x01ddcd (0x7bcd bytes) 000057 1787580812.509564 trunc from 0x029f5e to 0x039aaa (0xfb4d bytes) 000059 1787580812.520003 mapwrite 0x02cfe2 thru 0x030282 (0x32a1 bytes) fallocating to largest ever: 0x3e069 000060 1787580812.550760 fallocate 0x02f376 thru 0x03e068 (0xecf3 bytes) 000061 1787580812.555975 mapread 0x00e2c1 thru 0x015ac0 (0x7800 bytes) 000062 1787580812.563988 punch from 0x01d739 to 0x023770 (0x6038 bytes) 000063 1787580812.568890 trunc from 0x040000 to 0x00e9ef (0x31610 bytes) 000064 1787580812.587941 mapread 0x00be5e thru 0x00e9ef (0x2b92 bytes) 000065 1787580812.589404 mapwrite 0x017f57 thru 0x024ba9 (0xcc53 bytes) 000067 1787580812.652844 write 0x015632 thru 0x0206c3 (0xb092 bytes) 000068 1787580812.654946 mapread 0x00dbce thru 0x0147fc (0x6c2f bytes) 000069 1787580812.661468 fallocate 0x014584 thru 0x01e535 (0x9fb2 bytes) 000070 1787580812.666518 mapread 0x01e6b2 thru 0x0213e4 (0x2d33 bytes) 000071 1787580812.667696 fallocate 0x018306 thru 0x024764 (0xc45f bytes) 000072 1787580812.672772 write 0x021293 thru 0x021722 (0x490 bytes) 000073 1787580812.674613 mapread 0x01b91b thru 0x024ba9 (0x928f bytes) 000074 1787580812.675936 mapread 0x021e18 thru 0x024ba9 (0x2d92 bytes) 000075 1787580812.677076 mapread 0x0035a5 thru 0x011888 (0xe2e4 bytes) 000076 1787580812.685860 trunc from 0x024baa to 0x029859 (0x4cb0 bytes) 000077 1787580812.706414 mapwrite 0x021041 thru 0x02229b (0x125b bytes) 000079 1787580812.732715 trunc from 0x010bb9 to 0x0121e0 (0x1628 bytes) 000080 1787580812.738630 fallocate 0x0004af thru 0x00cc32 (0xc784 bytes) 000083 1787580812.746479 write 0x0089b9 thru 0x012062 (0x96aa bytes) 000084 1787580812.756071 read 0x0114ca thru 0x01e4c3 (0xcffa bytes) 000085 1787580812.756677 trunc from 0x0053d2 to 0x00d31a (0x7f49 bytes) 000086 1787580812.777438 fallocate 0x018459 thru 0x01f49d (0x7045 bytes) 000087 1787580812.784678 fallocate 0x021730 thru 0x023a82 (0x2353 bytes) 000088 1787580812.794822 punch from 0x027d73 to 0x0293c4 (0x1652 bytes) 000089 1787580812.802309 trunc from 0x013e21 to 0x022f5e (0xf13e bytes) 000090 1787580812.811059 read 0x002ea3 thru 0x012a6f (0xfbcd bytes) 000091 1787580812.830089 trunc from 0x02985a to 0x0225a8 (0x72b1 bytes) 000092 1787580812.848451 trunc from 0x019555 to 0x0225a8 (0x9054 bytes) 000093 1787580812.856319 trunc from 0x011e35 to 0x018b67 (0x6d33 bytes) 000094 1787580812.869001 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x225a9): Mirror 4: 0x8e8dee16 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x225a9): Mirror 4: 0x8e8dee16 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x225a9: skipped 000095 1787580813.185497 fallocate 0x0300e1 thru 0x03c2a5 (0xc1c5 bytes) 000096 1787580813.190028 read 0x0202f7 thru 0x0225a8 (0x22b2 bytes) 000097 1787580813.196861 mapwrite 0x01db14 thru 0x027439 (0x9926 bytes) 000098 1787580813.252509 fallocate 0x027f45 thru 0x02d2e6 (0x53a2 bytes) 000099 1787580813.256076 trunc from 0x02585d to 0x026b37 (0x12db bytes) 000101 1787580813.263566 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d2e7): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d2e7): Mirror 4: 0xd9dafc70 (stale) CRC-32 checksum value for chunk [0, 0x2d2e7): Mirror 1: 0xaef7c5bf Mirror 5: 0xaef7c5bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d2e7: skipped 000102 1787580814.107857 read 0x019790 thru 0x022bff (0x9470 bytes) 000103 1787580814.126284 trunc from 0x008c8e to 0x018b54 (0xfec7 bytes) 000104 1787580814.158704 read 0x026196 thru 0x02cbcf (0x6a3a bytes) 000105 1787580814.166006 trunc from 0x02d2e7 to 0x01fcd5 (0xd611 bytes) 000106 1787580814.201991 mapread 0x0146cd thru 0x01b0e3 (0x6a17 bytes) 000107 1787580814.203003 fallocate 0x025b61 thru 0x0319c4 (0xbe64 bytes) 000108 1787580814.206788 trunc from 0x0319c5 to 0x00e234 (0x23790 bytes) 000109 1787580814.218961 mapread 0x00b641 thru 0x00e234 (0x2bf4 bytes) 000110 1787580814.223867 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xe235): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe235): Mirror 4: 0x534a04f (stale) Mirror 5: 0xffc57493 (stale) CRC-32 checksum value for chunk [0, 0xe235): Mirror 1: 0x7e3b6767 Mirror 6: 0x7e3b6767 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe235: skipped 000111 1787580814.837516 trunc from 0x00e235 to 0x0235e3 (0x153af bytes) 000112 1787580814.919378 mapwrite 0x0267fa thru 0x03033f (0x9b46 bytes) 000113 1787580814.972106 trunc from 0x030340 to 0x023f17 (0xc428 bytes) 000114 1787580814.987473 write 0x003b28 thru 0x005f3b (0x2414 bytes) 000116 1787580814.997376 write 0x0213b7 thru 0x0294ab (0x80f5 bytes) 000117 1787580814.998795 mapread 0x019cfe thru 0x022b6b (0x8e6e bytes) 000118 1787580815.008186 read 0x026c0d thru 0x0294ab (0x289f bytes) 000119 1787580815.008831 fallocate 0x004882 thru 0x013b5f (0xf2de bytes) 000120 1787580815.013372 fallocate 0x025b7f thru 0x033aa8 (0xdf2a bytes) 000121 1787580815.021557 trunc from 0x0294ac to 0x03c202 (0x12d57 bytes) 000122 1787580815.045342 trunc from 0x03c203 to 0x01313f (0x290c3 bytes) 000123 1787580815.062003 mapwrite 0x025ad0 thru 0x027412 (0x1943 bytes) 000124 1787580815.097338 fallocate 0x01f626 thru 0x02112c (0x1b07 bytes) 000125 1787580815.101963 mapwrite 0x0043f4 thru 0x00a34e (0x5f5b bytes) 000126 1787580815.137593 mapread 0x013e99 thru 0x01fb64 (0xbccc bytes) 000127 1787580815.152484 mapread 0x0056b9 thru 0x014683 (0xefcb bytes) 000129 1787580815.154057 mapwrite 0x0260e3 thru 0x0302ef (0xa20d bytes) 000131 1787580815.212185 MIRROR_RESYNC 4 mirrors 000132 1787580816.030476 trunc from 0x0302f0 to 0x00f22a (0x210c5 bytes) 000133 1787580816.131927 trunc from 0x00c755 to 0x00f22a (0x2ad6 bytes) 000134 1787580816.139365 trunc from 0x00cb98 to 0x00f22a (0x2693 bytes) 000135 1787580816.161509 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xf22b): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf22b): Mirror 4: 0x95440951 (stale) Mirror 5: 0x95440951 (stale) Mirror 6: 0x95440951 (stale) CRC-32 checksum value for chunk [0, 0xf22b): Mirror 1: 0x95440951 Mirror 7: 0x95440951 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf22b: skipped 000136 1787580817.119039 fallocate 0x003b88 thru 0x00f54a (0xb9c3 bytes) 000138 1787580817.160207 trunc from 0x00f22b to 0x03bcfb (0x2cad1 bytes) 000139 1787580817.188612 fallocate 0x01b4fa thru 0x01c181 (0xc88 bytes) 000140 1787580817.193880 read 0x002f8e thru 0x012887 (0xf8fa bytes) 000141 1787580817.206564 trunc from 0x03bcfc to 0x0151f7 (0x26b04 bytes) 000142 1787580817.222659 trunc from 0x0151f8 to 0x039c38 (0x24a41 bytes) 000143 1787580817.235796 trunc from 0x00163a to 0x00d02c (0xb9f3 bytes) 000144 1787580817.258578 fallocate 0x03497b thru 0x034e2b (0x4b1 bytes) 000147 1787580817.273850 mapwrite 0x00938e thru 0x00d020 (0x3c93 bytes) 000148 1787580817.374328 fallocate 0x02fd73 thru 0x033797 (0x3a25 bytes) 000149 1787580817.379034 write 0x011f97 thru 0x0140eb (0x2155 bytes) 000150 1787580817.384453 mapwrite 0x0383b5 thru 0x03ffff (0x7c4b bytes) 000151 1787580817.469177 read 0x005380 thru 0x00a9a1 (0x5622 bytes) 000152 1787580817.469982 punch from 0x00844c to 0x008818 (0x3cd bytes) 000153 1787580817.480235 fallocate 0x020c2a thru 0x02146b (0x842 bytes) 000154 1787580817.496503 mapread 0x03b464 thru 0x03dab2 (0x264f bytes) 000155 1787580817.498002 mapread 0x012170 thru 0x01e33d (0xc1ce bytes) 000156 1787580817.514390 mapwrite 0x03f33b thru 0x03ffff (0xcc5 bytes) 000157 1787580817.560748 punch from 0x03cea3 to 0x03ffff (0x315d bytes) 000158 1787580817.572862 mapread 0x02ff1c thru 0x0358d0 (0x59b5 bytes) 000159 1787580817.573862 write 0x013aa2 thru 0x017be3 (0x4142 bytes) 000160 1787580817.585716 mapwrite 0x014e09 thru 0x01c4b6 (0x76ae bytes) 000161 1787580817.627952 fallocate 0x006de0 thru 0x0118e9 (0xab0a bytes) 000164 1787580817.645037 trunc from 0x040000 to 0x00e0d9 (0x31f26 bytes) 000165 1787580817.689990 trunc from 0x00ca20 to 0x00e0d9 (0x16ba bytes) 000166 1787580817.699103 trunc from 0x003a88 to 0x0075a3 (0x3b1c bytes) 000167 1787580817.703723 trunc from 0x00e0da to 0x024a6a (0x16991 bytes) 000168 1787580817.729363 trunc from 0x024a6b to 0x023ea8 (0xbc2 bytes) 000169 1787580817.774199 punch from 0x0031b6 to 0x0073a1 (0x41ec bytes) 000170 1787580817.784432 punch from 0x0011a7 to 0x00c88a (0xb6e4 bytes) 000172 1787580817.803294 trunc from 0x001087 to 0x00383b (0x27b5 bytes) 000173 1787580817.820681 fallocate 0x00ea37 thru 0x013dd7 (0x53a1 bytes) 000174 1787580817.833640 mapread 0x016ecd thru 0x01e555 (0x7689 bytes) 000175 1787580817.922415 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23ea9): Mirror 4: 0x4fe65313 (stale) Mirror 6: 0x4fe65313 (stale) Mirror 7: 0x4fe65313 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23ea9): Mirror 4: 0x4fe65313 (stale) Mirror 6: 0x4fe65313 (stale) Mirror 7: 0x4fe65313 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23ea9: skipped 000176 1787580818.492724 mapwrite 0x007a67 thru 0x010de4 (0x937e bytes) 000178 1787580818.543613 punch from 0x01f827 to 0x023ea8 (0x4682 bytes) 000179 1787580818.549512 trunc from 0x023ea9 to 0x0040c0 (0x1fde8 bytes) 000181 1787580818.632972 MIRROR_RESYNC 4 mirrors 000182 1787580819.497391 trunc from 0x001526 to 0x0040c0 (0x2b9b bytes) 000183 1787580819.542756 punch from 0x001947 to 0x0040c0 (0x277a bytes) 000185 1787580819.555739 punch from 0x003b05 to 0x0040c0 (0x5bc bytes) 000186 1787580819.561525 trunc from 0x0040c1 to 0x03140d (0x2d34d bytes) 000187 1787580819.588686 fallocate 0x031516 thru 0x039a69 (0x8554 bytes) 000189 1787580819.594131 write 0x01736c thru 0x02620f (0xeea4 bytes) 000190 1787580819.604760 trunc from 0x039a6a to 0x023f3a (0x15b2f bytes) 000191 1787580819.618403 mapread 0x004d23 thru 0x0141f2 (0xf4d0 bytes) 000192 1787580819.652630 write 0x0276c7 thru 0x036ead (0xf7e7 bytes) 000194 1787580819.655813 mapwrite 0x0106b2 thru 0x01e1a0 (0xdaef bytes) 000195 1787580819.674750 mapwrite 0x027ecc thru 0x028157 (0x28c bytes) 000196 1787580819.688214 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36eae): Mirror 6: 0xe9267566 (stale) Mirror 7: 0xe9267566 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36eae): Mirror 6: 0xe9267566 (stale) Mirror 7: 0xe9267566 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36eae: skipped 000197 1787580819.933290 mapwrite 0x0093dd thru 0x017d5a (0xe97e bytes) 000198 1787580819.980524 mapread 0x032b37 thru 0x035ea8 (0x3372 bytes) 000199 1787580819.990844 punch from 0x024bbc to 0x030d8f (0xc1d4 bytes) 000200 1787580819.996212 mapread 0x013ab8 thru 0x013ac6 (0xf bytes) 000201 1787580820.002634 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36eae): Mirror 7: 0xe9267566 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36eae): Mirror 7: 0xe9267566 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36eae: skipped 000202 1787580820.245120 fallocate 0x022f75 thru 0x02a8fb (0x7987 bytes) fallocating to largest ever: 0x3e8c9 000203 1787580820.251497 fallocate 0x035141 thru 0x03e8c8 (0x9788 bytes) 000204 1787580820.257398 mapwrite 0x01f442 thru 0x02c7a5 (0xd364 bytes) 000206 1787580820.302613 read 0x018a2b thru 0x01dbab (0x5181 bytes) 000207 1787580820.307303 trunc from 0x036eae to 0x00ba08 (0x2b4a5 bytes) 000208 1787580820.329730 punch from 0x00a96a to 0x00ba08 (0x109f bytes) 000209 1787580820.336859 mapwrite 0x0099c6 thru 0x01687c (0xceb7 bytes) 000210 1787580820.396697 read 0x000783 thru 0x009420 (0x8c9e bytes) 000211 1787580820.407349 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1687d): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1687d): Mirror 7: 0x869b44dd (stale) CRC-32 checksum value for chunk [0, 0x1687d): Mirror 1: 0x9b9714fa Mirror 8: 0x9b9714fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1687d: skipped 000213 1787580821.064825 read 0x00a603 thru 0x0127ba (0x81b8 bytes) fallocating to largest ever: 0x40000 000214 1787580821.102196 fallocate 0x03b5e9 thru 0x03ffff (0x4a17 bytes) 000215 1787580821.183628 write 0x014ec9 thru 0x01b802 (0x693a bytes) 000216 1787580821.215206 punch from 0x00d6a2 to 0x00f155 (0x1ab4 bytes) 000217 1787580821.220598 mapwrite 0x01da43 thru 0x024043 (0x6601 bytes) 000218 1787580821.353443 write 0x0337fc thru 0x03b267 (0x7a6c bytes) 000219 1787580821.355413 write 0x023b86 thru 0x024c12 (0x108d bytes) 000220 1787580821.365470 mapread 0x009680 thru 0x00d39f (0x3d20 bytes) 000221 1787580821.382178 punch from 0x0087f1 to 0x00bfda (0x37ea bytes) 000222 1787580821.409989 MIRROR_RESYNC 3 mirrors 000223 1787580822.626169 write 0x027272 thru 0x03206e (0xadfd bytes) 000224 1787580822.717241 mapread 0x02fc9a thru 0x03b267 (0xb5ce bytes) 000225 1787580822.758907 mapwrite 0x012a11 thru 0x01810e (0x56fe bytes) 000226 1787580822.861739 punch from 0x012278 to 0x0143f5 (0x217e bytes) 000227 1787580822.871472 mapwrite 0x004a25 thru 0x00d632 (0x8c0e bytes) 000228 1787580822.917371 trunc from 0x03b268 to 0x03a984 (0x8e3 bytes) 000230 1787580822.965088 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a985): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a985): Mirror 7: 0xefdab02f (stale) Mirror 8: 0xefdab02f (stale) CRC-32 checksum value for chunk [0, 0x3a985): Mirror 1: 0x74b3bb56 Mirror 9: 0x74b3bb56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a985: skipped 000231 1787580824.105177 trunc from 0x03a985 to 0x03a84d (0x137 bytes) 000232 1787580824.157450 fallocate 0x009db9 thru 0x015aeb (0xbd33 bytes) 000233 1787580824.163189 fallocate 0x022a98 thru 0x02ae49 (0x83b2 bytes) 000234 1787580824.169243 write 0x024826 thru 0x02cc78 (0x8453 bytes) 000235 1787580824.183173 MIRROR_RESYNC 4 mirrors 000236 1787580825.547422 trunc from 0x025719 to 0x028946 (0x322e bytes) 000238 1787580825.577135 trunc from 0x018707 to 0x022aaf (0xa3a9 bytes) 000240 1787580825.588175 trunc from 0x03a84e to 0x0322c5 (0x8588 bytes) 000241 1787580825.609375 fallocate 0x01fe58 thru 0x02a8b6 (0xaa5f bytes) 000242 1787580825.614199 punch from 0x0225aa to 0x02f539 (0xcf90 bytes) 000243 1787580825.619383 mapread 0x004476 thru 0x00e03e (0x9bc9 bytes) 000244 1787580825.635774 punch from 0x01d210 to 0x02a64e (0xd43f bytes) 000245 1787580825.644063 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x322c6): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x322c6): Mirror 7: 0x19a2f555 (stale) Mirror 8: 0x19a2f555 (stale) Mirror 9: 0x19a2f555 (stale) CRC-32 checksum value for chunk [0, 0x322c6): Mirror 1: 0x588c1a1a Mirror 10: 0x588c1a1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x322c6: skipped 000246 1787580826.782363 punch from 0x02c74e to 0x02e89f (0x2152 bytes) 000247 1787580826.832305 trunc from 0x00b196 to 0x0159da (0xa845 bytes) 000249 1787580826.841439 fallocate 0x034a7f thru 0x03f4b2 (0xaa34 bytes) 000250 1787580826.846497 punch from 0x02010c to 0x02e1e7 (0xe0dc bytes) 000251 1787580826.852380 fallocate 0x039062 thru 0x03ffff (0x6f9e bytes) 000252 1787580826.863832 trunc from 0x01ba54 to 0x0255a0 (0x9b4d bytes) 000253 1787580826.877632 trunc from 0x040000 to 0x038665 (0x799a bytes) 000254 1787580826.942720 trunc from 0x02c1e2 to 0x037546 (0xb365 bytes) 000256 1787580826.949170 mapread 0x02878e thru 0x030c44 (0x84b7 bytes) 000257 1787580827.007864 punch from 0x02970d to 0x036e1e (0xd712 bytes) 000259 1787580827.028091 mapwrite 0x01fae0 thru 0x02e76c (0xec8d bytes) 000260 1787580827.075214 mapread 0x011074 thru 0x020b8b (0xfb18 bytes) 000261 1787580827.083625 read 0x016524 thru 0x018cbc (0x2799 bytes) 000262 1787580827.084500 read 0x003269 thru 0x00fedc (0xcc74 bytes) 000263 1787580827.093087 punch from 0x02d077 to 0x0318fa (0x4884 bytes) 000264 1787580827.097180 mapread 0x017264 thru 0x02648e (0xf22b bytes) 000265 1787580827.098559 write 0x026ecf thru 0x02e649 (0x777b bytes) 000267 1787580827.104422 fallocate 0x023665 thru 0x02adc9 (0x7765 bytes) 000268 1787580827.112804 MIRROR_RESYNC 5 mirrors 000269 1787580829.081725 punch from 0x013fde to 0x01fc9f (0xbcc2 bytes) 000270 1787580829.132228 trunc from 0x038666 to 0x02c428 (0xc23d bytes) 000271 1787580829.158460 trunc from 0x02c429 to 0x019874 (0x12bb4 bytes) 000272 1787580829.179486 read 0x0177c7 thru 0x019874 (0x20ae bytes) 000273 1787580829.185168 mapread 0x00221a thru 0x00e152 (0xbf39 bytes) 000274 1787580829.202955 fallocate 0x036282 thru 0x0378c6 (0x1645 bytes) 000275 1787580829.208082 read 0x00dbb4 thru 0x0143d6 (0x6823 bytes) 000277 1787580829.221174 trunc from 0x0153a8 to 0x019874 (0x44cd bytes) 000278 1787580829.225777 write 0x026243 thru 0x02999c (0x375a bytes) 000279 1787580829.227424 mapread 0x008350 thru 0x0121f8 (0x9ea9 bytes) 000280 1787580829.229030 read 0x0104e2 thru 0x0203f7 (0xff16 bytes) 000281 1787580829.236615 trunc from 0x00d4c2 to 0x01800f (0xab4e bytes) 000282 1787580829.241946 write 0x03650b thru 0x03ffff (0x9af5 bytes) 000283 1787580829.244006 fallocate 0x0391da thru 0x03966a (0x491 bytes) 000284 1787580829.252697 MIRROR_RESYNC 5 mirrors 000285 1787580830.501123 mapread 0x01c8df thru 0x02167d (0x4d9f bytes) 000286 1787580830.533223 write 0x0012f1 thru 0x00899f (0x76af bytes) 000287 1787580830.580866 fallocate 0x0218c3 thru 0x02399c (0x20da bytes) 000289 1787580830.598209 trunc from 0x029390 to 0x03571d (0xc38e bytes) 000291 1787580830.616975 trunc from 0x01aead to 0x023de8 (0x8f3c bytes) 000292 1787580830.623684 mapwrite 0x001536 thru 0x00409a (0x2b65 bytes) 000293 1787580830.649334 punch from 0x01271e to 0x021449 (0xed2c bytes) 000294 1787580830.653907 read 0x024dbc thru 0x025416 (0x65b bytes) 000295 1787580830.658966 punch from 0x017f0d to 0x025caa (0xdd9e bytes) 000296 1787580830.664955 fallocate 0x02c060 thru 0x03325b (0x71fc bytes) 000297 1787580830.670371 write 0x01ddeb thru 0x027af4 (0x9d0a bytes) 000299 1787580830.699183 mapread 0x02900a thru 0x02ba3c (0x2a33 bytes) 000300 1787580830.718219 mapwrite 0x03ec63 thru 0x03ffff (0x139d bytes) 000301 1787580830.762764 write 0x002feb thru 0x0053ca (0x23e0 bytes) 000303 1787580830.764567 trunc from 0x02f550 to 0x0314e6 (0x1f97 bytes) 000304 1787580830.770138 punch from 0x01095c to 0x017b45 (0x71ea bytes) 000305 1787580830.775003 fallocate 0x02dd6e thru 0x0391b2 (0xb445 bytes) 000306 1787580830.781956 trunc from 0x040000 to 0x026106 (0x19ef9 bytes) 000307 1787580830.804362 mapwrite 0x00a2aa thru 0x017c76 (0xd9cd bytes) 000308 1787580830.849692 write 0x0289af thru 0x036646 (0xdc98 bytes) 000309 1787580830.858182 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36647): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36647): Mirror 7: 0xbb0e12ac (stale) Mirror 8: 0xbb0e12ac (stale) Mirror 9: 0xbb0e12ac (stale) Mirror 10: 0xbb0e12ac (stale) CRC-32 checksum value for chunk [0, 0x36647): Mirror 1: 0xdda42478 Mirror 11: 0xdda42478 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36647: skipped 000312 1787580831.888704 fallocate 0x02135b thru 0x030da1 (0xfa47 bytes) 000313 1787580831.927815 fallocate 0x014553 thru 0x01fdff (0xb8ad bytes) 000314 1787580831.935596 mapread 0x00572b thru 0x005f2f (0x805 bytes) 000315 1787580831.940790 read 0x013829 thru 0x01b4f2 (0x7cca bytes) 000316 1787580831.954738 mapwrite 0x022e84 thru 0x022e86 (0x3 bytes) 000317 1787580831.971999 trunc from 0x036647 to 0x014f91 (0x216b5 bytes) 000318 1787580832.003335 punch from 0x00fc64 to 0x014f91 (0x532e bytes) 000319 1787580832.013285 mapread 0x0140d1 thru 0x014f91 (0xec1 bytes) 000320 1787580832.024750 write 0x034528 thru 0x03e299 (0x9d72 bytes) 000321 1787580832.026692 trunc from 0x03e29a to 0x026777 (0x17b22 bytes) 000322 1787580832.050421 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26778): Mirror 8: 0xe22193b3 (stale) Mirror 9: 0xe22193b3 (stale) Mirror 10: 0xe22193b3 (stale) Mirror 11: 0x9d626f12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26778): Mirror 8: 0xe22193b3 (stale) Mirror 9: 0xe22193b3 (stale) Mirror 10: 0xe22193b3 (stale) Mirror 11: 0x9d626f12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26778: skipped 000323 1787580832.506762 write 0x027401 thru 0x037169 (0xfd69 bytes) 000324 1787580832.515078 mapread 0x025e19 thru 0x02669a (0x882 bytes) 000325 1787580832.524088 fallocate 0x016ac8 thru 0x01f36d (0x88a6 bytes) 000327 1787580832.529746 trunc from 0x00a604 to 0x0129cb (0x83c8 bytes) 000328 1787580832.535480 mapwrite 0x00c4c7 thru 0x0173c1 (0xaefb bytes) 000329 1787580832.578414 read 0x00d768 thru 0x0141c6 (0x6a5f bytes) 000330 1787580832.579404 trunc from 0x015d5e to 0x020c20 (0xaec3 bytes) 000331 1787580832.585498 trunc from 0x03716a to 0x01689e (0x208cb bytes) 000332 1787580832.598298 read 0x0072e8 thru 0x007cac (0x9c5 bytes) 000333 1787580832.599312 trunc from 0x005f60 to 0x005f83 (0x24 bytes) 000334 1787580832.605227 mapread 0x002ee3 thru 0x00a302 (0x7420 bytes) 000336 1787580832.620580 trunc from 0x01689f to 0x00daad (0x8df1 bytes) 000338 1787580832.640241 write 0x03dd83 thru 0x03ffff (0x227d bytes) 000339 1787580832.642506 trunc from 0x02c01e to 0x0380be (0xc0a1 bytes) 000340 1787580832.647525 fallocate 0x0286a3 thru 0x036a3e (0xe39c bytes) 000341 1787580832.652091 trunc from 0x040000 to 0x007e1f (0x381e0 bytes) 000343 1787580832.668026 trunc from 0x007cdc to 0x007e1f (0x144 bytes) 000344 1787580832.674665 read 0x004e8a thru 0x007e1f (0x2f96 bytes) 000345 1787580832.680043 mapwrite 0x03dc1a thru 0x03ffff (0x23e6 bytes) 000346 1787580832.726402 MIRROR_RESYNC 5 mirrors 000347 1787580833.646166 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000348 1787580833.982464 trunc from 0x040000 to 0x0278d5 (0x1872a bytes) 000349 1787580834.033894 mapwrite 0x027eec thru 0x031c27 (0x9d3c bytes) 000350 1787580834.087649 read 0x00d483 thru 0x0176c6 (0xa244 bytes) 000351 1787580834.096138 fallocate 0x000b0d thru 0x00fa5b (0xef4f bytes) 000352 1787580834.111810 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31c28): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c28): Mirror 8: 0x72dfba42 (stale) Mirror 10: 0x72dfba42 (stale) Mirror 11: 0x72dfba42 (stale) CRC-32 checksum value for chunk [0, 0x31c28): Mirror 1: 0x8aae51b6 Mirror 12: 0x8aae51b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c28: skipped 000353 1787580834.840006 fallocate 0x010481 thru 0x018a59 (0x85d9 bytes) 000355 1787580834.879403 mapread 0x00de4e thru 0x018e92 (0xb045 bytes) 000356 1787580834.904590 write 0x011182 thru 0x01d3d7 (0xc256 bytes) 000357 1787580834.910440 write 0x029da6 thru 0x037513 (0xd76e bytes) 000358 1787580834.919179 mapread 0x0051bd thru 0x006fab (0x1def bytes) 000359 1787580834.929150 mapread 0x011917 thru 0x012e0b (0x14f5 bytes) 000360 1787580834.929986 punch from 0x00e891 to 0x01cf79 (0xe6e9 bytes) 000361 1787580834.945124 read 0x011272 thru 0x017eee (0x6c7d bytes) 000362 1787580834.955226 punch from 0x007e50 to 0x00a5bc (0x276d bytes) 000363 1787580834.960586 fallocate 0x001cd3 thru 0x0087d2 (0x6b00 bytes) 000364 1787580834.965636 punch from 0x02fa56 to 0x037513 (0x7abe bytes) 000365 1787580834.970867 trunc from 0x037514 to 0x002409 (0x3510a bytes) 000366 1787580835.016042 trunc from 0x00240a to 0x025d2b (0x23922 bytes) 000367 1787580835.036541 punch from 0x0236d8 to 0x025d2b (0x2654 bytes) 000368 1787580835.041999 fallocate 0x004c14 thru 0x01162b (0xca18 bytes) 000369 1787580835.048079 trunc from 0x025d2c to 0x002f38 (0x22df3 bytes) 000370 1787580835.064001 mapread 0x0003e8 thru 0x002f38 (0x2b51 bytes) 000371 1787580835.071449 punch from 0x002f02 to 0x002f38 (0x37 bytes) 000372 1787580835.077323 trunc from 0x002f39 to 0x022a8f (0x1fb57 bytes) 000373 1787580835.099949 write 0x013908 thru 0x0230ca (0xf7c3 bytes) 000374 1787580835.111763 MIRROR_RESYNC 5 mirrors 000375 1787580835.928332 punch from 0x019dd6 to 0x021260 (0x748b bytes) 000376 1787580835.962100 mapread 0x00c27c thru 0x00d0a8 (0xe2d bytes) 000377 1787580835.973061 trunc from 0x00cd14 to 0x011046 (0x4333 bytes) 000378 1787580835.978068 write 0x034b55 thru 0x039af5 (0x4fa1 bytes) 000379 1787580835.980246 trunc from 0x002b1b to 0x00da23 (0xaf09 bytes) 000380 1787580835.986004 mapwrite 0x032083 thru 0x03626c (0x41ea bytes) 000381 1787580836.009470 fallocate 0x018f6a thru 0x01a3e1 (0x1478 bytes) 000382 1787580836.013989 trunc from 0x02cba5 to 0x038be1 (0xc03d bytes) 000383 1787580836.019183 read 0x02562c thru 0x0354e8 (0xfebd bytes) 000385 1787580836.031081 mapread 0x00fdc6 thru 0x01dcf3 (0xdf2e bytes) 000387 1787580836.050615 punch from 0x036367 to 0x037640 (0x12da bytes) 000388 1787580836.055948 mapwrite 0x03d5cd thru 0x03ffff (0x2a33 bytes) 000389 1787580836.102577 mapwrite 0x014c28 thru 0x023c3b (0xf014 bytes) 000390 1787580836.132280 mapread 0x038640 thru 0x03c284 (0x3c45 bytes) 000391 1787580836.133301 write 0x00772f thru 0x00cbab (0x547d bytes) 000392 1787580836.139945 mapread 0x03d18f thru 0x03ffff (0x2e71 bytes) 000393 1787580836.141070 write 0x02038e thru 0x02a7cf (0xa442 bytes) 000394 1787580836.143280 mapread 0x0028d1 thru 0x0124aa (0xfbda bytes) 000396 1787580836.154387 fallocate 0x007b2c thru 0x00b49d (0x3972 bytes) 000397 1787580836.159951 mapwrite 0x019b94 thru 0x01c297 (0x2704 bytes) 000398 1787580836.177553 trunc from 0x040000 to 0x0107ee (0x2f811 bytes) 000399 1787580836.205999 MIRROR_RESYNC 5 mirrors 000400 1787580836.931806 mapwrite 0x031b8a thru 0x036f55 (0x53cc bytes) 000401 1787580837.027461 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x36f56): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f56): Mirror 8: 0x15c55e92 (stale) Mirror 10: 0x15c55e92 (stale) Mirror 11: 0x15c55e92 (stale) Mirror 12: 0x15c55e92 (stale) CRC-32 checksum value for chunk [0, 0x36f56): Mirror 1: 0xf84834fd Mirror 13: 0xf84834fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f56: skipped 000402 1787580837.868656 fallocate 0x01f6b9 thru 0x029206 (0x9b4e bytes) 000403 1787580837.935959 trunc from 0x036f56 to 0x0134e3 (0x23a72 bytes) 000404 1787580837.971159 punch from 0x010d26 to 0x0134e3 (0x27be bytes) 000405 1787580837.978832 trunc from 0x0134e4 to 0x01f10c (0xbc29 bytes) 000406 1787580838.005192 mapwrite 0x023425 thru 0x031081 (0xdc5d bytes) 000407 1787580838.097811 MIRROR_RESYNC 6 mirrors 000408 1787580839.776468 punch from 0x01858a to 0x025eae (0xd925 bytes) 000409 1787580839.816771 write 0x007a27 thru 0x00fdcc (0x83a6 bytes) 000410 1787580839.826532 trunc from 0x031082 to 0x00debf (0x231c2 bytes) 000411 1787580839.847534 mapwrite 0x0019c0 thru 0x00db91 (0xc1d2 bytes) 000412 1787580839.884922 trunc from 0x002901 to 0x009a6f (0x716f bytes) 000413 1787580839.889730 punch from 0x006ff5 to 0x00debf (0x6ecb bytes) 000414 1787580839.895401 write 0x01c02a thru 0x01c2a6 (0x27d bytes) 000415 1787580839.896834 read 0x01aa62 thru 0x01c2a6 (0x1845 bytes) 000416 1787580839.902369 punch from 0x0181c3 to 0x01b439 (0x3277 bytes) 000417 1787580839.907189 punch from 0x00266d to 0x008e95 (0x6829 bytes) 000418 1787580839.913218 trunc from 0x01c2a7 to 0x033e14 (0x17b6e bytes) 000420 1787580839.931994 read 0x02fb3f thru 0x033e14 (0x42d6 bytes) 000421 1787580839.944089 mapread 0x01f7f0 thru 0x026d37 (0x7548 bytes) 000423 1787580839.945788 trunc from 0x01f321 to 0x027982 (0x8662 bytes) 000424 1787580839.951320 write 0x00e44f thru 0x01227e (0x3e30 bytes) 000425 1787580839.953387 fallocate 0x0098b4 thru 0x01518a (0xb8d7 bytes) 000426 1787580839.962428 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33e15): Mirror 10: 0x460432b1 (stale) Mirror 11: 0x460432b1 (stale) Mirror 12: 0x460432b1 (stale) Mirror 13: 0x460432b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e15): Mirror 10: 0x460432b1 (stale) Mirror 11: 0x460432b1 (stale) Mirror 12: 0x460432b1 (stale) Mirror 13: 0x460432b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e15: skipped 000427 1787580840.355290 mapwrite 0x00cac9 thru 0x01a643 (0xdb7b bytes) 000429 1787580840.408011 trunc from 0x033e15 to 0x00dcd6 (0x2613e bytes) 000430 1787580840.434711 trunc from 0x00ba1a to 0x00dcd6 (0x22bd bytes) 000431 1787580840.439593 write 0x034ba0 thru 0x03ffff (0xb460 bytes) 000432 1787580840.442132 punch from 0x01daed to 0x01ff6e (0x2482 bytes) 000433 1787580840.447212 mapread 0x0046a3 thru 0x00b5cd (0x6f2b bytes) 000434 1787580840.459517 punch from 0x02b090 to 0x02dac8 (0x2a39 bytes) 000435 1787580840.466023 trunc from 0x02b2dc to 0x03b12e (0xfe53 bytes) 000436 1787580840.480781 mapread 0x0038ec thru 0x007aa4 (0x41b9 bytes) 000437 1787580840.481852 trunc from 0x040000 to 0x02c73c (0x138c3 bytes) 000439 1787580840.498367 punch from 0x0177ee to 0x025e29 (0xe63c bytes) 000440 1787580840.503085 trunc from 0x01e10d to 0x026383 (0x8277 bytes) 000441 1787580840.508187 mapwrite 0x024abb thru 0x02ea79 (0x9fbf bytes) 000444 1787580840.564416 read 0x01f5e9 thru 0x027623 (0x803b bytes) 000445 1787580840.566131 trunc from 0x02ea7a to 0x0247ad (0xa2cc bytes) 000446 1787580840.589343 write 0x01fc5e thru 0x02c354 (0xc6f7 bytes) 000447 1787580840.592114 fallocate 0x01a4e2 thru 0x022bc2 (0x86e1 bytes) 000448 1787580840.598406 mapread 0x01e8dd thru 0x021460 (0x2b84 bytes) 000449 1787580840.600209 trunc from 0x02c355 to 0x017efa (0x1445a bytes) 000450 1787580840.631864 MIRROR_RESYNC 5 mirrors 000451 1787580841.551468 write 0x018c55 thru 0x019ed2 (0x127e bytes) 000452 1787580841.587329 fallocate 0x036c3b thru 0x03ffff (0x93c5 bytes) 000453 1787580841.603478 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19ed3): Mirror 10: 0x72163baa (stale) Mirror 11: 0x72163baa (stale) Mirror 12: 0x72163baa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19ed3): Mirror 10: 0x72163baa (stale) Mirror 11: 0x72163baa (stale) Mirror 12: 0x72163baa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19ed3: skipped 000454 1787580841.929355 write 0x017f27 thru 0x024b41 (0xcc1b bytes) 000455 1787580841.940181 write 0x0119dd thru 0x012160 (0x784 bytes) 000456 1787580841.949245 trunc from 0x024b42 to 0x013333 (0x1180e bytes) 000457 1787580841.966986 fallocate 0x0186f9 thru 0x01cd33 (0x463b bytes) 000458 1787580841.972510 punch from 0x002eff to 0x00b5f0 (0x86f2 bytes) 000460 1787580841.978710 read 0x00c1e1 thru 0x011f6c (0x5d8c bytes) 000461 1787580841.985535 trunc from 0x013334 to 0x022ec4 (0xfb91 bytes) 000462 1787580842.002797 trunc from 0x022ec5 to 0x028657 (0x5793 bytes) 000463 1787580842.018047 trunc from 0x01330f to 0x01bc27 (0x8919 bytes) 000464 1787580842.023413 trunc from 0x028658 to 0x01450b (0x1414c bytes) 000465 1787580842.035135 trunc from 0x0063cc to 0x013897 (0xd4cc bytes) 000466 1787580842.043249 mapwrite 0x023cf7 thru 0x03173d (0xda47 bytes) 000467 1787580842.100615 MIRROR_RESYNC 4 mirrors 000469 1787580843.074067 MIRROR_EXTEND to 5 mirrors 000470 1787580843.643992 punch from 0x016e08 to 0x01a1ab (0x33a4 bytes) 000471 1787580843.688712 fallocate 0x0283dc thru 0x035e3c (0xda61 bytes) 000472 1787580843.696373 write 0x01f653 thru 0x029906 (0xa2b4 bytes) 000473 1787580843.716963 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x35e3d): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e3d): Mirror 10: 0x36ab1b55 (stale) Mirror 11: 0x36ab1b55 (stale) Mirror 12: 0x36ab1b55 (stale) Mirror 13: 0x36ab1b55 (stale) CRC-32 checksum value for chunk [0, 0x35e3d): Mirror 1: 0x9ae31a02 Mirror 14: 0x9ae31a02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e3d: skipped 000474 1787580844.549067 trunc from 0x030805 to 0x035e3c (0x5638 bytes) 000475 1787580844.599854 MIRROR_RESYNC 6 mirrors 000476 1787580845.799015 trunc from 0x01cf75 to 0x02b40f (0xe49b bytes) 000477 1787580845.834857 trunc from 0x022424 to 0x02d44b (0xb028 bytes) 000478 1787580845.838892 mapwrite 0x02372c thru 0x028048 (0x491d bytes) 000479 1787580845.870289 mapread 0x0180bf thru 0x02754a (0xf48c bytes) 000481 1787580845.880703 mapwrite 0x03b7fc thru 0x03f2c4 (0x3ac9 bytes) 000482 1787580845.919847 write 0x0022e1 thru 0x010d15 (0xea35 bytes) 000483 1787580845.930187 mapwrite 0x03372f thru 0x038bad (0x547f bytes) 000485 1787580845.966766 punch from 0x00564e to 0x013914 (0xe2c7 bytes) 000487 1787580845.978355 fallocate 0x0175bb thru 0x02663b (0xf081 bytes) 000488 1787580845.983985 write 0x018d90 thru 0x01b588 (0x27f9 bytes) 000489 1787580845.985429 punch from 0x00480e to 0x006d71 (0x2564 bytes) 000490 1787580845.990873 trunc from 0x03f2c5 to 0x002b35 (0x3c78f bytes) 000491 1787580846.011174 write 0x031e96 thru 0x0325fb (0x766 bytes) 000493 1787580846.014080 trunc from 0x0325fc to 0x001b13 (0x30ae8 bytes) 000494 1787580846.031813 fallocate 0x00c516 thru 0x01c1f4 (0xfcdf bytes) 000495 1787580846.037531 trunc from 0x001b14 to 0x01d1fc (0x1b6e9 bytes) 000496 1787580846.053978 mapread 0x017e7b thru 0x01d1fc (0x5382 bytes) 000497 1787580846.062709 read 0x019c10 thru 0x01d1fc (0x35ed bytes) 000499 1787580846.064746 mapwrite 0x00008b thru 0x006f57 (0x6ecd bytes) 000500 1787580846.087658 trunc from 0x01977b to 0x01d1fc (0x3a82 bytes) 000501 1787580846.092106 fallocate 0x012d48 thru 0x018a5b (0x5d14 bytes) 000503 1787580846.096791 read 0x00259b thru 0x008f6c (0x69d2 bytes) 000504 1787580846.101107 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d1fd): Mirror 11: 0x47fa2549 (stale) Mirror 12: 0x47fa2549 (stale) Mirror 13: 0x47fa2549 (stale) Mirror 14: 0x47fa2549 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d1fd): Mirror 11: 0x47fa2549 (stale) Mirror 12: 0x47fa2549 (stale) Mirror 13: 0x47fa2549 (stale) Mirror 14: 0x47fa2549 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d1fd: skipped 000505 1787580846.398158 trunc from 0x01d1fd to 0x03bf0f (0x1ed13 bytes) 000506 1787580846.414604 fallocate 0x03b8e3 thru 0x03ffff (0x471d bytes) 000507 1787580846.420768 write 0x000573 thru 0x00d2e7 (0xcd75 bytes) 000508 1787580846.430349 punch from 0x03aecf to 0x03ffff (0x5131 bytes) 000509 1787580846.433985 trunc from 0x011a25 to 0x01f68b (0xdc67 bytes) 000510 1787580846.438899 mapwrite 0x03a220 thru 0x03eb08 (0x48e9 bytes) 000511 1787580846.474414 fallocate 0x014319 thru 0x018964 (0x464c bytes) 000512 1787580846.482126 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x8c9bb622 (stale) Mirror 12: 0x8c9bb622 (stale) Mirror 13: 0x8c9bb622 (stale) Mirror 14: 0x8c9bb622 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4ddc3e16 Mirror 15: 0x4ddc3e16 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000513 1787580847.120234 mapread 0x036a63 thru 0x0375d5 (0xb73 bytes) 000514 1787580847.145826 MIRROR_RESYNC 6 mirrors 000515 1787580848.126901 read 0x00dd20 thru 0x019f3a (0xc21b bytes) 000516 1787580848.135513 mapread 0x02a63d thru 0x02c5b9 (0x1f7d bytes) 000517 1787580848.162047 MIRROR_RESYNC 6 mirrors 000518 1787580848.799014 punch from 0x03502e to 0x03872f (0x3702 bytes) 000520 1787580848.872015 write 0x001f34 thru 0x0076a5 (0x5772 bytes) 000521 1787580848.885124 MIRROR_RESYNC 6 mirrors 000522 1787580850.346635 trunc from 0x029613 to 0x037be8 (0xe5d6 bytes) 000523 1787580850.390898 mapwrite 0x0254d9 thru 0x02fb22 (0xa64a bytes) 000524 1787580850.453474 write 0x00c9b4 thru 0x00db77 (0x11c4 bytes) 000526 1787580850.471404 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x83cba1da (stale) Mirror 12: 0x83cba1da (stale) Mirror 13: 0x83cba1da (stale) Mirror 14: 0x83cba1da (stale) Mirror 15: 0x83cba1da (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe8ac700c Mirror 16: 0xe8ac700c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000528 1787580851.546064 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x83cba1da (stale) Mirror 12: 0x83cba1da (stale) Mirror 13: 0x83cba1da (stale) Mirror 15: 0x83cba1da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0x83cba1da (stale) Mirror 12: 0x83cba1da (stale) Mirror 13: 0x83cba1da (stale) Mirror 15: 0x83cba1da (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe8ac700c Mirror 16: 0xe8ac700c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000529 1787580852.479816 trunc from 0x040000 to 0x027f33 (0x180cc bytes) 000530 1787580852.544978 trunc from 0x0162e1 to 0x024822 (0xe542 bytes) 000531 1787580852.555719 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27f34): Mirror 12: 0x459ccec (stale) Mirror 13: 0x459ccec (stale) Mirror 15: 0x459ccec (stale) Mirror 16: 0x5e546c8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f34): Mirror 12: 0x459ccec (stale) Mirror 13: 0x459ccec (stale) Mirror 15: 0x459ccec (stale) Mirror 16: 0x5e546c8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f34: skipped 000532 1787580852.981188 trunc from 0x005cad to 0x0105f1 (0xa945 bytes) 000533 1787580852.989835 read 0x008e5b thru 0x00cdbb (0x3f61 bytes) 000534 1787580852.999099 trunc from 0x009b8c to 0x014a07 (0xae7c bytes) 000535 1787580853.005991 mapread 0x01071a thru 0x010e67 (0x74e bytes) 000536 1787580853.015981 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27f34): Mirror 12: 0x459ccec (stale) Mirror 13: 0x459ccec (stale) Mirror 16: 0x5e546c8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f34): Mirror 12: 0x459ccec (stale) Mirror 13: 0x459ccec (stale) Mirror 16: 0x5e546c8f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f34: skipped 000537 1787580853.300382 mapwrite 0x018e6f thru 0x020d00 (0x7e92 bytes) 000539 1787580853.360074 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x27f34): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f34): Mirror 12: 0x459ccec (stale) Mirror 13: 0x459ccec (stale) Mirror 16: 0x5e546c8f (stale) CRC-32 checksum value for chunk [0, 0x27f34): Mirror 1: 0x2e88f50 Mirror 17: 0x2e88f50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f34: skipped 000540 1787580854.306795 fallocate 0x01cccc thru 0x021270 (0x45a5 bytes) 000541 1787580854.352219 mapread 0x003046 thru 0x00ecd8 (0xbc93 bytes) 000542 1787580854.369015 trunc from 0x01ee4b to 0x027f33 (0x90e9 bytes) 000543 1787580854.374334 fallocate 0x02a1a4 thru 0x02d782 (0x35df bytes) 000544 1787580854.379586 trunc from 0x02d783 to 0x017b88 (0x15bfa bytes) 000545 1787580854.433154 punch from 0x015b12 to 0x015bac (0x9b bytes) 000547 1787580854.440201 read 0x0044e3 thru 0x004c85 (0x7a3 bytes) 000548 1787580854.442853 mapread 0x002032 thru 0x00ae69 (0x8e38 bytes) 000549 1787580854.444066 write 0x0073fd thru 0x016acf (0xf6d3 bytes) 000550 1787580854.450985 mapread 0x000ae6 thru 0x003d62 (0x327d bytes) 000551 1787580854.456187 mapwrite 0x001634 thru 0x00d1ae (0xbb7b bytes) 000552 1787580854.486452 MIRROR_RESYNC 5 mirrors 000553 1787580855.439829 trunc from 0x00a560 to 0x0150f9 (0xab9a bytes) 000554 1787580855.493687 read 0x0165d7 thru 0x017b88 (0x15b2 bytes) 000555 1787580855.513236 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x17b89): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17b89): Mirror 12: 0xf193a829 (stale) Mirror 13: 0xf193a829 (stale) Mirror 16: 0xf193a829 (stale) Mirror 17: 0xf193a829 (stale) CRC-32 checksum value for chunk [0, 0x17b89): Mirror 1: 0xecfc6c2d Mirror 18: 0xecfc6c2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17b89: skipped 000556 1787580856.789095 trunc from 0x00be2b to 0x0139d8 (0x7bae bytes) 000557 1787580856.822776 trunc from 0x017b89 to 0x01fdbf (0x8237 bytes) 000558 1787580856.856815 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fdc0): Mirror 12: 0xe1f9aad3 (stale) Mirror 13: 0xe1f9aad3 (stale) Mirror 16: 0xe1f9aad3 (stale) Mirror 17: 0xe1f9aad3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fdc0): Mirror 12: 0xe1f9aad3 (stale) Mirror 13: 0xe1f9aad3 (stale) Mirror 16: 0xe1f9aad3 (stale) Mirror 17: 0xe1f9aad3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fdc0: skipped 000559 1787580857.329024 read 0x01a72e thru 0x01f0d1 (0x49a4 bytes) 000560 1787580857.345161 punch from 0x017913 to 0x01fdbf (0x84ad bytes) 000561 1787580857.356777 trunc from 0x011d0f to 0x01cec1 (0xb1b3 bytes) 000562 1787580857.362825 mapread 0x015b16 thru 0x01fdbf (0xa2aa bytes) 000563 1787580857.386865 punch from 0x01f652 to 0x01fdbf (0x76e bytes) 000564 1787580857.392022 write 0x0359d1 thru 0x039761 (0x3d91 bytes) 000565 1787580857.398083 fallocate 0x03bd31 thru 0x03ffff (0x42cf bytes) 000566 1787580857.404926 fallocate 0x00e83d thru 0x015130 (0x68f4 bytes) 000567 1787580857.411771 fallocate 0x0037bc thru 0x013269 (0xfaae bytes) 000570 1787580857.419998 trunc from 0x0310ab to 0x03815d (0x70b3 bytes) 000571 1787580857.440998 trunc from 0x00de4c to 0x01bcde (0xde93 bytes) 000572 1787580857.447782 fallocate 0x022000 thru 0x024311 (0x2312 bytes) 000573 1787580857.453594 punch from 0x02eedf to 0x02fa32 (0xb54 bytes) 000574 1787580857.458841 mapwrite 0x00d156 thru 0x01952c (0xc3d7 bytes) 000575 1787580857.522898 trunc from 0x039762 to 0x002bb4 (0x36bad bytes) 000576 1787580857.545368 fallocate 0x035cd2 thru 0x03f030 (0x935f bytes) 000577 1787580857.551834 fallocate 0x01c6c1 thru 0x020aa0 (0x43e0 bytes) 000578 1787580857.557147 read 0x0011a5 thru 0x002bb4 (0x1a10 bytes) 000579 1787580857.562775 read 0x002303 thru 0x002bb4 (0x8b2 bytes) 000581 1787580857.563714 read 0x002b92 thru 0x002bb4 (0x23 bytes) 000582 1787580857.569128 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2bb5): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bb5): Mirror 12: 0x6fe52432 (stale) Mirror 13: 0x6fe52432 (stale) Mirror 16: 0x6fe52432 (stale) Mirror 17: 0x6fe52432 (stale) CRC-32 checksum value for chunk [0, 0x2bb5): Mirror 1: 0x6fe52432 Mirror 18: 0x6fe52432 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bb5: skipped 000584 1787580858.652840 fallocate 0x006018 thru 0x013453 (0xd43c bytes) 000585 1787580858.767891 read 0x00234f thru 0x002bb4 (0x866 bytes) 000586 1787580858.776881 mapwrite 0x010965 thru 0x01fb8a (0xf226 bytes) 000587 1787580858.927517 punch from 0x013376 to 0x01b755 (0x83e0 bytes) 000588 1787580858.935240 trunc from 0x01bc28 to 0x01fb8a (0x3f63 bytes) 000589 1787580858.943492 fallocate 0x00ed4a thru 0x011211 (0x24c8 bytes) 000590 1787580858.950754 trunc from 0x01fb8b to 0x00d4d2 (0x126b8 bytes) 000591 1787580858.986036 read 0x005a36 thru 0x009249 (0x3814 bytes) 000592 1787580859.003349 write 0x0187ec thru 0x027eec (0xf701 bytes) 000594 1787580859.005701 mapread 0x026805 thru 0x027143 (0x93f bytes) 000595 1787580859.008588 trunc from 0x00b94e to 0x0174d8 (0xbb8b bytes) 000596 1787580859.015828 mapread 0x0207b2 thru 0x027eec (0x773b bytes) 000597 1787580859.018852 mapwrite 0x03f68e thru 0x03ffff (0x972 bytes) 000598 1787580859.144187 MIRROR_RESYNC 6 mirrors 000599 1787580861.750571 mapread 0x00149f thru 0x002941 (0x14a3 bytes) 000600 1787580861.758817 trunc from 0x040000 to 0x02d224 (0x12ddb bytes) 000602 1787580861.860031 trunc from 0x0074dc to 0x01095e (0x9483 bytes) 000603 1787580861.876690 read 0x0178fd thru 0x01cb41 (0x5245 bytes) 000604 1787580861.893725 MIRROR_RESYNC 6 mirrors 000606 1787580863.562140 mapread 0x01b37a thru 0x02aa24 (0xf6ab bytes) 000607 1787580863.586720 fallocate 0x0280cd thru 0x02dffd (0x5f31 bytes) 000608 1787580863.640190 write 0x00cdec thru 0x0147ff (0x7a14 bytes) 000609 1787580863.649920 trunc from 0x02dffe to 0x02753c (0x6ac1 bytes) 000610 1787580863.691554 trunc from 0x01985e to 0x02753c (0xdcdf bytes) 000611 1787580863.699303 mapread 0x00a17e thru 0x01233c (0x81bf bytes) 000612 1787580863.742311 punch from 0x010b7e to 0x0164fc (0x597f bytes) 000613 1787580863.810540 mapwrite 0x03cde2 thru 0x03ffff (0x321e bytes) 000614 1787580863.906564 fallocate 0x005acc thru 0x01463f (0xeb74 bytes) 000615 1787580863.921038 fallocate 0x022021 thru 0x022539 (0x519 bytes) 000616 1787580863.933075 trunc from 0x0046b6 to 0x009fe4 (0x592f bytes) 000617 1787580863.956545 mapwrite 0x001686 thru 0x010110 (0xea8b bytes) 000618 1787580864.043328 trunc from 0x040000 to 0x026838 (0x197c7 bytes) 000619 1787580864.058525 mapwrite 0x0035c1 thru 0x004121 (0xb61 bytes) 000620 1787580864.072991 mapwrite 0x02eda6 thru 0x032663 (0x38be bytes) 000621 1787580864.183960 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32664): Mirror 12: 0x8a070792 (stale) Mirror 13: 0x8a070792 (stale) Mirror 17: 0x8a070792 (stale) Mirror 18: 0x8a070792 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32664): Mirror 12: 0x8a070792 (stale) Mirror 13: 0x8a070792 (stale) Mirror 17: 0x8a070792 (stale) Mirror 18: 0x8a070792 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32664: skipped 000622 1787580864.796008 mapwrite 0x013037 thru 0x01d93e (0xa908 bytes) 000623 1787580864.849520 read 0x004c28 thru 0x00c884 (0x7c5d bytes) 000624 1787580864.858729 punch from 0x010574 to 0x0202bb (0xfd48 bytes) 000625 1787580864.874056 punch from 0x0139d6 to 0x016c9b (0x32c6 bytes) 000626 1787580864.880498 fallocate 0x005c86 thru 0x01200f (0xc38a bytes) 000628 1787580864.886422 fallocate 0x007e8f thru 0x0124b5 (0xa627 bytes) 000629 1787580864.893026 read 0x006cce thru 0x011ad7 (0xae0a bytes) 000630 1787580864.933787 mapread 0x008d5b thru 0x00ba7a (0x2d20 bytes) 000631 1787580864.934809 mapwrite 0x031826 thru 0x03c6e8 (0xaec3 bytes) 000632 1787580864.987673 punch from 0x00a967 to 0x013ea9 (0x9543 bytes) 000633 1787580864.993108 write 0x034772 thru 0x03da03 (0x9292 bytes) 000634 1787580864.995770 punch from 0x0210a8 to 0x022339 (0x1292 bytes) 000635 1787580864.999697 trunc from 0x03da04 to 0x004170 (0x39893 bytes) 000636 1787580865.023733 trunc from 0x004171 to 0x00139b (0x2dd5 bytes) 000637 1787580865.041747 punch from 0x000ca6 to 0x00139b (0x6f6 bytes) 000639 1787580865.047044 punch from 0x000d00 to 0x00139b (0x69c bytes) 000641 1787580865.056655 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x139c): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x139c): Mirror 12: 0x9fdbf5f9 (stale) Mirror 13: 0x9fdbf5f9 (stale) Mirror 17: 0x9fdbf5f9 (stale) Mirror 18: 0x9fdbf5f9 (stale) CRC-32 checksum value for chunk [0, 0x139c): Mirror 1: 0x57b45c36 Mirror 19: 0x57b45c36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x139c: skipped 000642 1787580865.815359 trunc from 0x000383 to 0x00139b (0x1019 bytes) 000643 1787580865.855654 read 0x001036 thru 0x00139b (0x366 bytes) 000644 1787580865.861734 read 0x000f00 thru 0x00139b (0x49c bytes) 000647 1787580865.867363 trunc from 0x00139c to 0x00c988 (0xb5ed bytes) 000648 1787580865.888499 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc989): Mirror 13: 0x6df7f4e8 (stale) Mirror 17: 0x6df7f4e8 (stale) Mirror 18: 0x6df7f4e8 (stale) Mirror 19: 0xc20ca68e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc989): Mirror 13: 0x6df7f4e8 (stale) Mirror 17: 0x6df7f4e8 (stale) Mirror 18: 0x6df7f4e8 (stale) Mirror 19: 0xc20ca68e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc989: skipped 000649 1787580866.239739 mapread 0x00880e thru 0x00ba62 (0x3255 bytes) 000650 1787580866.261433 trunc from 0x00c989 to 0x00a52a (0x245e bytes) 000651 1787580866.279481 mapread 0x002470 thru 0x003799 (0x132a bytes) 000652 1787580866.286078 trunc from 0x00223b to 0x00a52a (0x82f0 bytes) 000653 1787580866.291483 mapread 0x001b17 thru 0x009b23 (0x800d bytes) 000654 1787580866.305498 mapwrite 0x021097 thru 0x024632 (0x359c bytes) 000656 1787580866.361361 fallocate 0x0014ba thru 0x0101f9 (0xed40 bytes) 000657 1787580866.376702 mapwrite 0x03aa87 thru 0x03ffff (0x5579 bytes) 000658 1787580866.457530 MIRROR_RESYNC 5 mirrors 000659 1787580867.875367 mapwrite 0x036664 thru 0x03ffff (0x999c bytes) 000660 1787580868.002178 write 0x03fa0a thru 0x03ffff (0x5f6 bytes) 000661 1787580868.005781 punch from 0x007e5a to 0x00eec4 (0x706b bytes) 000662 1787580868.018294 read 0x030bb8 thru 0x038db5 (0x81fe bytes) 000663 1787580868.019131 fallocate 0x035380 thru 0x0370c3 (0x1d44 bytes) 000664 1787580868.035465 write 0x01bb6b thru 0x02a651 (0xeae7 bytes) 000665 1787580868.071072 MIRROR_RESYNC 5 mirrors 000667 1787580869.603599 write 0x02d1ac thru 0x03930d (0xc162 bytes) 000668 1787580869.652031 read 0x0104c8 thru 0x0122cb (0x1e04 bytes) 000669 1787580869.660100 fallocate 0x00c8b0 thru 0x01817c (0xb8cd bytes) 000670 1787580869.665141 write 0x03573f thru 0x03ffff (0xa8c1 bytes) 000671 1787580869.667226 trunc from 0x01e26f to 0x026b0d (0x889f bytes) 000672 1787580869.678407 punch from 0x0230e9 to 0x02ec68 (0xbb80 bytes) 000673 1787580869.708559 write 0x039a8e thru 0x03ffff (0x6572 bytes) 000674 1787580869.709965 fallocate 0x005076 thru 0x0138c7 (0xe852 bytes) 000675 1787580869.717799 read 0x02a6ac thru 0x031086 (0x69db bytes) 000676 1787580869.734243 punch from 0x00fbf0 to 0x0133ed (0x37fe bytes) 000677 1787580869.738554 write 0x011c48 thru 0x0214c2 (0xf87b bytes) 000678 1787580869.752447 fallocate 0x02aa44 thru 0x035636 (0xabf3 bytes) 000679 1787580869.759426 mapwrite 0x034c55 thru 0x0362e8 (0x1694 bytes) 000680 1787580869.777250 write 0x01ad2d thru 0x01f566 (0x483a bytes) 000681 1787580869.779454 write 0x00f454 thru 0x01c24c (0xcdf9 bytes) 000682 1787580869.785050 punch from 0x038f37 to 0x03ffff (0x70c9 bytes) 000683 1787580869.791371 mapread 0x0210d9 thru 0x025d95 (0x4cbd bytes) 000684 1787580869.793155 fallocate 0x038ef8 thru 0x03ffff (0x7108 bytes) 000685 1787580869.807522 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x997dc898 (stale) Mirror 17: 0x997dc898 (stale) Mirror 19: 0x997dc898 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x997dc898 (stale) Mirror 17: 0x997dc898 (stale) Mirror 19: 0x997dc898 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000686 1787580870.112390 trunc from 0x03c88b to 0x03ffff (0x3775 bytes) 000687 1787580870.117365 read 0x0270ea thru 0x027a84 (0x99b bytes) 000688 1787580870.122623 mapread 0x03f22a thru 0x03ffff (0xdd6 bytes) 000689 1787580870.131391 read 0x013f59 thru 0x018671 (0x4719 bytes) 000690 1787580870.154005 fallocate 0x010aca thru 0x017e51 (0x7388 bytes) 000691 1787580870.159965 punch from 0x03a8d9 to 0x03ffff (0x5727 bytes) 000692 1787580870.165716 trunc from 0x02d263 to 0x03a01d (0xcdbb bytes) 000693 1787580870.175604 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x997dc898 (stale) Mirror 17: 0x997dc898 (stale) Mirror 19: 0x997dc898 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2f186bc1 Mirror 20: 0x2f186bc1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000694 1787580871.347033 trunc from 0x022c71 to 0x030c97 (0xe027 bytes) 000696 1787580871.382990 punch from 0x0345e9 to 0x037e98 (0x38b0 bytes) 000698 1787580871.387686 mapwrite 0x0296e8 thru 0x037605 (0xdf1e bytes) 000699 1787580871.450629 read 0x01b6ca thru 0x0244a1 (0x8dd8 bytes) 000700 1787580871.460448 write 0x01bdf7 thru 0x01fee7 (0x40f1 bytes) 000701 1787580871.470986 mapread 0x02f20c thru 0x038d90 (0x9b85 bytes) 000702 1787580871.473605 trunc from 0x019934 to 0x026ec3 (0xd590 bytes) 000703 1787580871.496204 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x997dc898 (stale) Mirror 17: 0x997dc898 (stale) Mirror 19: 0x997dc898 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x997dc898 (stale) Mirror 17: 0x997dc898 (stale) Mirror 19: 0x997dc898 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000704 1787580871.805392 fallocate 0x0233ca thru 0x02cc39 (0x9870 bytes) 000705 1787580871.814541 MIRROR_RESYNC 4 mirrors 000706 1787580872.970781 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000707 1787580873.233778 read 0x02b506 thru 0x035b8e (0xa689 bytes) 000708 1787580873.243642 punch from 0x0223d9 to 0x02de1a (0xba42 bytes) 000709 1787580873.278500 mapwrite 0x02eba5 thru 0x038039 (0x9495 bytes) 000710 1787580873.327547 read 0x027eba thru 0x033ff2 (0xc139 bytes) 000711 1787580873.336588 trunc from 0x040000 to 0x0252e2 (0x1ad1d bytes) 000712 1787580873.361858 trunc from 0x0252e3 to 0x02571f (0x43d bytes) 000713 1787580873.378960 MIRROR_RESYNC 3 mirrors 000714 1787580874.171889 fallocate 0x009498 thru 0x017a51 (0xe5ba bytes) 000715 1787580874.230475 mapwrite 0x019b18 thru 0x0251b1 (0xb69a bytes) 000716 1787580874.276530 mapread 0x01bea5 thru 0x023d8e (0x7eea bytes) 000717 1787580874.282589 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x25720): Mirror 13: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25720): Mirror 13: 0xc87d92dc (stale) Mirror 19: 0xc87d92dc (stale) CRC-32 checksum value for chunk [0, 0x25720): Mirror 1: 0x19998fa4 Mirror 20: 0x19998fa4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25720: skipped 000718 1787580875.280146 read 0x0092c0 thru 0x0096d8 (0x419 bytes) 000720 1787580875.322088 mapwrite 0x034c41 thru 0x039127 (0x44e7 bytes) 000722 1787580875.450347 read 0x00a0e6 thru 0x00aa8e (0x9a9 bytes) 000723 1787580875.486881 mapread 0x0022c2 thru 0x00630e (0x404d bytes) 000724 1787580875.489189 mapread 0x02d2e4 thru 0x037504 (0xa221 bytes) 000725 1787580875.490519 mapread 0x016db9 thru 0x01789c (0xae4 bytes) 000726 1787580875.496320 trunc from 0x039128 to 0x016c31 (0x224f6 bytes) 000727 1787580875.569616 write 0x03f8ac thru 0x03ffff (0x754 bytes) 000728 1787580875.570673 fallocate 0x02ad22 thru 0x03ab93 (0xfe72 bytes) 000729 1787580875.578848 write 0x0232cc thru 0x02e784 (0xb4b9 bytes) 000730 1787580875.598139 write 0x0220fc thru 0x026692 (0x4597 bytes) 000731 1787580875.605573 fallocate 0x00fc29 thru 0x01700f (0x73e7 bytes) 000732 1787580875.631739 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x76d0b5fa (stale) Mirror 20: 0xd8f0d945 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x76d0b5fa (stale) Mirror 20: 0xd8f0d945 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000734 1787580875.993149 read 0x038244 thru 0x03ffff (0x7dbc bytes) 000735 1787580876.005850 punch from 0x006ea1 to 0x00714a (0x2aa bytes) 000736 1787580876.009082 write 0x01384b thru 0x01e8b6 (0xb06c bytes) 000738 1787580876.021124 trunc from 0x03baab to 0x03ffff (0x4555 bytes) 000739 1787580876.026278 trunc from 0x00fd4e to 0x019249 (0x94fc bytes) 000740 1787580876.039721 trunc from 0x040000 to 0x0275f7 (0x18a08 bytes) 000741 1787580876.056445 read 0x00e66e thru 0x00ff3b (0x18ce bytes) 000742 1787580876.064515 mapread 0x026b7a thru 0x0275f7 (0xa7e bytes) 000744 1787580876.065869 read 0x026508 thru 0x0275f7 (0x10f0 bytes) 000745 1787580876.066373 fallocate 0x03ecd1 thru 0x03ffff (0x132f bytes) 000746 1787580876.072181 trunc from 0x018afe to 0x01b577 (0x2a7a bytes) 000747 1787580876.077233 mapread 0x01f6f0 thru 0x0275f7 (0x7f08 bytes) 000748 1787580876.078892 mapwrite 0x02401e thru 0x027abc (0x3a9f bytes) 000749 1787580876.110980 mapread 0x01408e thru 0x014633 (0x5a6 bytes) 000750 1787580876.114779 fallocate 0x02c7df thru 0x03868a (0xbeac bytes) 000751 1787580876.120559 trunc from 0x026b6f to 0x02a6f8 (0x3b8a bytes) 000752 1787580876.125405 punch from 0x0212be to 0x025bd4 (0x4917 bytes) 000753 1787580876.129911 write 0x03c061 thru 0x03ffff (0x3f9f bytes) 000754 1787580876.131474 punch from 0x02c30e to 0x03940f (0xd102 bytes) 000755 1787580876.135968 read 0x0029e6 thru 0x004d1a (0x2335 bytes) 000756 1787580876.137139 write 0x0360bc thru 0x03ffff (0x9f44 bytes) 000757 1787580876.157902 fallocate 0x0029f2 thru 0x00567d (0x2c8c bytes) 000759 1787580876.163181 trunc from 0x040000 to 0x02ffc0 (0x1003f bytes) 000760 1787580876.179172 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ffc1): Mirror 20: 0x33be2671 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffc1): Mirror 20: 0x33be2671 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffc1: skipped 000762 1787580876.428435 punch from 0x00676c to 0x00782d (0x10c2 bytes) 000763 1787580876.433772 trunc from 0x02ffc1 to 0x035dc4 (0x5e04 bytes) 000764 1787580876.455982 trunc from 0x035dc5 to 0x02911d (0xcca7 bytes) 000765 1787580876.480515 fallocate 0x00b22c thru 0x00e39a (0x316f bytes) 000766 1787580876.489932 mapread 0x01b3fe thru 0x0269cd (0xb5d0 bytes) 000767 1787580876.533168 trunc from 0x02911e to 0x037fa7 (0xee8a bytes) 000768 1787580876.548484 fallocate 0x00285c thru 0x00357a (0xd1f bytes) 000769 1787580876.564451 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37fa8): Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fa8): Mirror 20: 0xf5193e2a (stale) CRC-32 checksum value for chunk [0, 0x37fa8): Mirror 1: 0x903d6202 Mirror 21: 0x903d6202 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fa8: skipped 000770 1787580877.228712 mapwrite 0x033f2a thru 0x035180 (0x1257 bytes) 000771 1787580877.352713 trunc from 0x011c54 to 0x011d3e (0xeb bytes) 000772 1787580877.362536 mapwrite 0x02f0f1 thru 0x03c6d5 (0xd5e5 bytes) 000773 1787580877.514790 read 0x026040 thru 0x035fb8 (0xff79 bytes) 000774 1787580877.527668 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c6d6): Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6d6): Mirror 20: 0x7706038a (stale) Mirror 21: 0x72115d50 (stale) CRC-32 checksum value for chunk [0, 0x3c6d6): Mirror 1: 0x84e15e40 Mirror 22: 0x84e15e40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6d6: skipped 000775 1787580878.327153 fallocate 0x023ba8 thru 0x02fe87 (0xc2e0 bytes) 000776 1787580878.365923 MIRROR_RESYNC 4 mirrors 000777 1787580879.363810 read 0x0241f2 thru 0x024771 (0x580 bytes) 000778 1787580879.368662 mapwrite 0x00a3e5 thru 0x0103a3 (0x5fbf bytes) 000779 1787580879.444779 trunc from 0x03c6d6 to 0x03ee17 (0x2742 bytes) 000780 1787580879.467371 punch from 0x027165 to 0x032c06 (0xbaa2 bytes) 000781 1787580879.472496 write 0x002073 thru 0x003df2 (0x1d80 bytes) 000782 1787580879.482888 mapwrite 0x02731e thru 0x03484d (0xd530 bytes) 000783 1787580879.514862 read 0x01cd5e thru 0x01e051 (0x12f4 bytes) 000784 1787580879.529545 trunc from 0x03ee18 to 0x01ce41 (0x21fd6 bytes) 000785 1787580879.545687 punch from 0x0146f6 to 0x01ce41 (0x874c bytes) 000786 1787580879.552056 fallocate 0x01e241 thru 0x028adc (0xa89c bytes) 000787 1787580879.555848 mapwrite 0x031a60 thru 0x03a45b (0x89fc bytes) 000788 1787580879.623122 trunc from 0x03a45c to 0x032a41 (0x7a1a bytes) 000790 1787580879.638302 mapwrite 0x01da25 thru 0x025cdb (0x82b7 bytes) 000791 1787580879.696828 trunc from 0x02b569 to 0x02e39a (0x2e32 bytes) 000792 1787580879.702375 mapwrite 0x028a6f thru 0x0380ce (0xf660 bytes) 000793 1787580879.812433 read 0x001c66 thru 0x005b63 (0x3efe bytes) 000794 1787580879.813335 write 0x0223c4 thru 0x02c96b (0xa5a8 bytes) 000795 1787580879.816195 write 0x003f44 thru 0x010993 (0xca50 bytes) 000796 1787580879.817828 punch from 0x0189ed to 0x01e805 (0x5e19 bytes) 000797 1787580879.824466 mapread 0x018704 thru 0x026d88 (0xe685 bytes) 000800 1787580879.838412 read 0x02b7b8 thru 0x02b9c3 (0x20c bytes) 000801 1787580879.839198 trunc from 0x030ebc to 0x0380ce (0x7213 bytes) 000802 1787580879.844860 punch from 0x009ad2 to 0x00afa9 (0x14d8 bytes) 000804 1787580879.860553 mapread 0x012351 thru 0x01f663 (0xd313 bytes) 000805 1787580879.862552 punch from 0x01bb42 to 0x01d8f4 (0x1db3 bytes) 000806 1787580879.867767 trunc from 0x0380cf to 0x017368 (0x20d66 bytes) 000807 1787580879.889857 write 0x028cbe thru 0x03207c (0x93bf bytes) 000808 1787580879.891757 mapwrite 0x00d0b7 thru 0x01b2e4 (0xe22e bytes) 000809 1787580879.934344 read 0x0285b9 thru 0x02e071 (0x5ab9 bytes) 000810 1787580879.935924 punch from 0x00e610 to 0x015f64 (0x7955 bytes) 000812 1787580879.942314 trunc from 0x03207d to 0x032329 (0x2ad bytes) 000813 1787580879.960352 punch from 0x0061e2 to 0x010a84 (0xa8a3 bytes) 000814 1787580879.966398 trunc from 0x00c4b2 to 0x00e271 (0x1dc0 bytes) 000815 1787580879.970895 trunc from 0x02a156 to 0x032329 (0x81d4 bytes) 000816 1787580879.976342 mapread 0x0057ad thru 0x00a634 (0x4e88 bytes) 000817 1787580879.997869 punch from 0x02827a to 0x032329 (0xa0b0 bytes) 000818 1787580880.003156 mapwrite 0x01d02d thru 0x01ed0f (0x1ce3 bytes) 000819 1787580880.015332 mapread 0x0209a6 thru 0x02b918 (0xaf73 bytes) 000820 1787580880.038571 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3232a): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3232a): Mirror 20: 0x6a6e6962 (stale) Mirror 21: 0x6a6e6962 (stale) Mirror 22: 0x6a6e6962 (stale) CRC-32 checksum value for chunk [0, 0x3232a): Mirror 1: 0x4962a22c Mirror 23: 0x4962a22c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3232a: skipped 000821 1787580881.030809 read 0x00430e thru 0x00c9b2 (0x86a5 bytes) 000822 1787580881.052896 trunc from 0x001c44 to 0x008027 (0x63e4 bytes) 000823 1787580881.123194 MIRROR_RESYNC 5 mirrors 000824 1787580882.607926 write 0x038e40 thru 0x03af1e (0x20df bytes) 000825 1787580882.661051 write 0x022fc9 thru 0x02c68b (0x96c3 bytes) 000826 1787580882.685519 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af1f): Mirror 21: 0x88cd73a (stale) Mirror 22: 0x88cd73a (stale) Mirror 23: 0x88cd73a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af1f): Mirror 21: 0x88cd73a (stale) Mirror 22: 0x88cd73a (stale) Mirror 23: 0x88cd73a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af1f: skipped 000827 1787580883.035058 mapread 0x006eea thru 0x00c60a (0x5721 bytes) 000828 1787580883.059621 mapread 0x03a8fb thru 0x03af1e (0x624 bytes) 000829 1787580883.068977 fallocate 0x016bc9 thru 0x01fb2b (0x8f63 bytes) 000830 1787580883.076139 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3af1f): Mirror 21: 0x88cd73a (stale) Mirror 22: 0x88cd73a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af1f): Mirror 21: 0x88cd73a (stale) Mirror 22: 0x88cd73a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af1f: skipped 000832 1787580883.315068 trunc from 0x023c32 to 0x027702 (0x3ad1 bytes) 000833 1787580883.319783 fallocate 0x014bc5 thru 0x01ae04 (0x6240 bytes) 000834 1787580883.331123 MIRROR_RESYNC 3 mirrors 000835 1787580884.362459 punch from 0x00c244 to 0x0182ec (0xc0a9 bytes) 000836 1787580884.389217 punch from 0x01e257 to 0x028053 (0x9dfd bytes) 000837 1787580884.394306 read 0x0219c4 thru 0x0240b4 (0x26f1 bytes) 000838 1787580884.400673 fallocate 0x01d7eb thru 0x01e44f (0xc65 bytes) 000839 1787580884.405836 write 0x0051c6 thru 0x0116e4 (0xc51f bytes) 000840 1787580884.415133 fallocate 0x027b9a thru 0x02e8a3 (0x6d0a bytes) 000841 1787580884.419712 punch from 0x01bf7c to 0x027ad3 (0xbb58 bytes) 000842 1787580884.424404 trunc from 0x000b9c to 0x005229 (0x468e bytes) 000843 1787580884.437693 mapread 0x00c014 thru 0x013022 (0x700f bytes) 000846 1787580884.448605 punch from 0x026d7a to 0x02aa98 (0x3d1f bytes) 000847 1787580884.453166 read 0x00342a thru 0x00aba2 (0x7779 bytes) 000848 1787580884.474708 write 0x005a12 thru 0x00d9fc (0x7feb bytes) 000849 1787580884.479127 MIRROR_RESYNC 3 mirrors 000850 1787580885.627546 mapread 0x02b7ea thru 0x02ca45 (0x125c bytes) 000851 1787580885.640148 read 0x01c9d4 thru 0x0232db (0x6908 bytes) 000852 1787580885.648987 mapwrite 0x03e9e0 thru 0x03ffff (0x1620 bytes) 000853 1787580885.699227 mapwrite 0x038848 thru 0x03ffff (0x77b8 bytes) 000854 1787580885.714716 trunc from 0x01e9aa to 0x02bc4c (0xd2a3 bytes) 000855 1787580885.719124 trunc from 0x035537 to 0x03eb97 (0x9661 bytes) 000856 1787580885.724153 read 0x0085b7 thru 0x012d5e (0xa7a8 bytes) 000858 1787580885.735136 trunc from 0x01d9fc to 0x01e2cb (0x8d0 bytes) 000859 1787580885.738713 read 0x00c244 thru 0x011b8c (0x5949 bytes) 000860 1787580885.739477 mapread 0x0125cf thru 0x0171b6 (0x4be8 bytes) 000861 1787580885.740484 read 0x02fbc1 thru 0x03e975 (0xedb5 bytes) 000862 1787580885.741701 mapread 0x019408 thru 0x01a16e (0xd67 bytes) 000863 1787580885.742797 mapwrite 0x003bb9 thru 0x012d0a (0xf152 bytes) 000864 1787580885.763148 mapwrite 0x02a2e8 thru 0x02b5bb (0x12d4 bytes) 000865 1787580885.777709 mapwrite 0x00a1af thru 0x01759a (0xd3ec bytes) 000866 1787580885.796855 write 0x024cc5 thru 0x02d161 (0x849d bytes) 000867 1787580885.797988 mapwrite 0x00b9bf thru 0x016099 (0xa6db bytes) 000868 1787580885.815570 punch from 0x021620 to 0x02fdae (0xe78f bytes) 000869 1787580885.820530 read 0x00e5c2 thru 0x013053 (0x4a92 bytes) 000870 1787580885.821193 mapread 0x03ef88 thru 0x03ffff (0x1078 bytes) 000871 1787580885.822497 fallocate 0x036ca2 thru 0x03f444 (0x87a3 bytes) 000872 1787580885.828054 trunc from 0x040000 to 0x013aa7 (0x2c558 bytes) 000873 1787580885.849864 mapwrite 0x023ff8 thru 0x03344b (0xf454 bytes) 000874 1787580885.894192 read 0x02d108 thru 0x03344b (0x6344 bytes) 000875 1787580885.895405 punch from 0x00ca14 to 0x017e75 (0xb462 bytes) 000876 1787580885.906212 fallocate 0x0120a0 thru 0x01ff2b (0xde8c bytes) 000877 1787580885.912477 read 0x0141bd thru 0x0240fc (0xff40 bytes) 000879 1787580885.931805 trunc from 0x03344c to 0x0268fc (0xcb4f bytes) 000881 1787580885.950007 read 0x024da1 thru 0x0268fc (0x1b5c bytes) 000882 1787580885.950594 fallocate 0x021862 thru 0x02e214 (0xc9b3 bytes) 000883 1787580885.957046 trunc from 0x00249f to 0x01219b (0xfcfd bytes) 000884 1787580885.961636 punch from 0x020c60 to 0x02cde1 (0xc182 bytes) 000885 1787580885.967001 mapread 0x01b974 thru 0x027f01 (0xc58e bytes) 000887 1787580885.983248 trunc from 0x00e435 to 0x01e408 (0xffd4 bytes) 000889 1787580885.990671 write 0x03dcce thru 0x03ffff (0x2332 bytes) 000890 1787580885.991602 trunc from 0x040000 to 0x009841 (0x367be bytes) 000891 1787580886.020929 trunc from 0x009842 to 0x01f28c (0x15a4b bytes) 000892 1787580886.046525 trunc from 0x01c4fd to 0x01f28c (0x2d90 bytes) 000893 1787580886.055323 trunc from 0x01f28d to 0x0327b6 (0x1352a bytes) 000895 1787580886.073469 fallocate 0x0102cf thru 0x014be4 (0x4916 bytes) 000896 1787580886.081813 mapread 0x013fac thru 0x017a3c (0x3a91 bytes) 000897 1787580886.113468 mapwrite 0x0094da thru 0x00fced (0x6814 bytes) 000898 1787580886.157932 punch from 0x027796 to 0x02d364 (0x5bcf bytes) 000899 1787580886.163777 mapread 0x01a5df thru 0x023f0c (0x992e bytes) 000901 1787580886.212026 trunc from 0x0327b7 to 0x010a9c (0x21d1a bytes) 000902 1787580886.228188 read 0x001f08 thru 0x00e94f (0xca48 bytes) 000903 1787580886.234498 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10a9d): Mirror 21: 0x55924e52 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10a9d): Mirror 21: 0x55924e52 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10a9d: skipped 000904 1787580886.506128 mapwrite 0x02e844 thru 0x0310b4 (0x2871 bytes) 000905 1787580886.562016 mapread 0x0268ff thru 0x0310b4 (0xa7b6 bytes) 000906 1787580886.570217 trunc from 0x018a17 to 0x02301c (0xa606 bytes) 000907 1787580886.576726 write 0x026f58 thru 0x029742 (0x27eb bytes) 000908 1787580886.582345 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000909 1787580886.727153 read 0x002834 thru 0x00c561 (0x9d2e bytes) 000910 1787580886.737531 trunc from 0x0189ad to 0x01db39 (0x518d bytes) 000911 1787580886.747069 fallocate 0x028260 thru 0x02da1c (0x57bd bytes) 000913 1787580886.758792 MIRROR_EXTEND to 2 mirrors 000914 1787580887.878717 read 0x00080b thru 0x006c53 (0x6449 bytes) 000915 1787580887.892657 fallocate 0x007582 thru 0x01744e (0xfecd bytes) 000916 1787580887.941150 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000917 1787580888.064126 mapwrite 0x0329eb thru 0x03ffff (0xd615 bytes) 000919 1787580888.123077 mapwrite 0x03d978 thru 0x03ffff (0x2688 bytes) 000920 1787580888.135336 mapread 0x025a84 thru 0x02f27d (0x97fa bytes) 000921 1787580888.145195 fallocate 0x00d906 thru 0x01c53f (0xec3a bytes) 000922 1787580888.150107 punch from 0x036172 to 0x03ffff (0x9e8e bytes) 000923 1787580888.155031 read 0x02edd9 thru 0x03a302 (0xb52a bytes) 000924 1787580888.162634 fallocate 0x0312b9 thru 0x03444a (0x3192 bytes) 000925 1787580888.167008 trunc from 0x040000 to 0x0088f9 (0x37706 bytes) 000927 1787580888.180033 write 0x003e09 thru 0x01181d (0xda15 bytes) 000928 1787580888.186728 write 0x01eb8d thru 0x0297d6 (0xac4a bytes) 000929 1787580888.190153 punch from 0x016fad to 0x01f86d (0x88c1 bytes) 000930 1787580888.215493 mapwrite 0x0322b6 thru 0x039be7 (0x7932 bytes) 000931 1787580888.261981 MIRROR_EXTEND to 2 mirrors 000932 1787580888.872348 write 0x01f3ff thru 0x02d34a (0xdf4c bytes) 000933 1787580888.923179 mapwrite 0x00bd80 thru 0x0109cd (0x4c4e bytes) 000935 1787580888.961179 mapread 0x036612 thru 0x039be7 (0x35d6 bytes) 000936 1787580888.971698 fallocate 0x0138e1 thru 0x014d9f (0x14bf bytes) 000937 1787580888.981104 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39be8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39be8): Mirror 2: 0xa5e4e3e2 (stale) CRC-32 checksum value for chunk [0, 0x39be8): Mirror 1: 0xc0da3a92 Mirror 3: 0xc0da3a92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39be8: skipped 000938 1787580889.721937 MIRROR_RESYNC 3 mirrors 000939 1787580890.388695 read 0x00fa43 thru 0x016d2a (0x72e8 bytes) 000940 1787580890.399589 punch from 0x03331f to 0x03811a (0x4dfc bytes) 000941 1787580890.442418 trunc from 0x039be8 to 0x016a77 (0x23170 bytes) 000942 1787580890.463325 mapread 0x0078f1 thru 0x013b09 (0xc219 bytes) 000943 1787580890.488679 punch from 0x000073 to 0x00752e (0x74bc bytes) 000944 1787580890.493672 mapread 0x00acac thru 0x00fadf (0x4e34 bytes) 000946 1787580890.495722 mapread 0x00b7c5 thru 0x01695c (0xb198 bytes) 000947 1787580890.497238 trunc from 0x016a78 to 0x025101 (0xe68a bytes) 000948 1787580890.510331 punch from 0x01a66c to 0x022ebc (0x8851 bytes) 000949 1787580890.515074 trunc from 0x01ffd8 to 0x025101 (0x512a bytes) 000950 1787580890.523173 MIRROR_RESYNC 3 mirrors 000951 1787580891.259321 trunc from 0x025102 to 0x03bb15 (0x16a14 bytes) 000952 1787580891.312482 mapread 0x0001e0 thru 0x00fe74 (0xfc95 bytes) 000953 1787580891.355022 trunc from 0x01c506 to 0x01e036 (0x1b31 bytes) 000954 1787580891.360933 mapread 0x01b9f9 thru 0x01d3fd (0x1a05 bytes) 000955 1787580891.383430 mapwrite 0x013632 thru 0x0138db (0x2aa bytes) 000956 1787580891.422680 punch from 0x02ff10 to 0x033814 (0x3905 bytes) 000957 1787580891.440831 mapwrite 0x02b583 thru 0x035ccb (0xa749 bytes) 000958 1787580891.539478 trunc from 0x00cee3 to 0x014d08 (0x7e26 bytes) 000959 1787580891.547429 trunc from 0x0177de to 0x0212c4 (0x9ae7 bytes) 000960 1787580891.564222 trunc from 0x02403d to 0x0296ce (0x5692 bytes) 000961 1787580891.581967 write 0x00d6e5 thru 0x011249 (0x3b65 bytes) 000962 1787580891.600953 trunc from 0x03bb16 to 0x02cbce (0xef47 bytes) 000964 1787580891.636415 write 0x039fad thru 0x03e2e5 (0x4339 bytes) 000965 1787580891.637634 punch from 0x02eae2 to 0x02eeee (0x40d bytes) 000966 1787580891.661606 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e2e6): Mirror 3: 0xf96f9454 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e2e6): Mirror 3: 0xf96f9454 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e2e6: skipped 000967 1787580892.094836 trunc from 0x03e2e6 to 0x03f842 (0x155d bytes) 000968 1787580892.112955 mapread 0x02cdda thru 0x0303e9 (0x3610 bytes) 000969 1787580892.125923 fallocate 0x00b062 thru 0x00f96f (0x490e bytes) 000970 1787580892.131638 write 0x0290af thru 0x02fee0 (0x6e32 bytes) 000971 1787580892.133046 trunc from 0x00c359 to 0x013694 (0x733c bytes) 000972 1787580892.137738 mapwrite 0x019655 thru 0x022531 (0x8edd bytes) 000973 1787580892.166364 mapread 0x01fec8 thru 0x020aa6 (0xbdf bytes) 000974 1787580892.168151 trunc from 0x00718f to 0x00f75d (0x85cf bytes) 000975 1787580892.178897 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) 000977 1787580892.337653 mapread 0x0079c0 thru 0x00d73c (0x5d7d bytes) 000978 1787580892.364563 MIRROR_EXTEND to 2 mirrors 000979 1787580893.213365 MIRROR_EXTEND to 3 mirrors 000980 1787580894.030126 MIRROR_EXTEND to 4 mirrors 000981 1787580894.761233 mapread 0x004612 thru 0x0075ad (0x2f9c bytes) 000982 1787580894.787182 punch from 0x027392 to 0x0353f1 (0xe060 bytes) 000983 1787580894.837990 read 0x00f40c thru 0x01d27f (0xde74 bytes) 000984 1787580894.848705 fallocate 0x00f485 thru 0x010fee (0x1b6a bytes) 000986 1787580894.857788 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f843): Mirror 3: 0x879433c5 (stale) Mirror 4: 0x879433c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f843): Mirror 3: 0x879433c5 (stale) Mirror 4: 0x879433c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f843: skipped 000987 1787580895.142177 mapwrite 0x017a40 thru 0x01c987 (0x4f48 bytes) 000988 1787580895.173800 trunc from 0x01f3e0 to 0x02ac7a (0xb89b bytes) 000989 1787580895.179382 read 0x007910 thru 0x00b94b (0x403c bytes) 000990 1787580895.191042 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f843): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f843): Mirror 3: 0x879433c5 (stale) Mirror 4: 0x879433c5 (stale) CRC-32 checksum value for chunk [0, 0x3f843): Mirror 1: 0xc43a8073 Mirror 5: 0xc43a8073 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f843: skipped 000991 1787580896.277903 mapread 0x0387ab thru 0x03f842 (0x7098 bytes) 000992 1787580896.339188 read 0x021ea3 thru 0x030fa7 (0xf105 bytes) 000993 1787580896.364888 trunc from 0x037d20 to 0x03d435 (0x5716 bytes) 000994 1787580896.470425 mapwrite 0x03189a thru 0x03af82 (0x96e9 bytes) 000995 1787580896.581398 trunc from 0x03f843 to 0x015063 (0x2a7df bytes) 000996 1787580896.638784 trunc from 0x011ce5 to 0x01213b (0x457 bytes) 000997 1787580896.644094 read 0x00c2a8 thru 0x010ecd (0x4c26 bytes) 000998 1787580896.652851 write 0x03bec3 thru 0x03ffff (0x413d bytes) 000999 1787580896.654032 trunc from 0x002351 to 0x00c228 (0x9ed8 bytes) 001000 1787580896.662624 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x43:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x98ea46ce (stale) Mirror 4: 0x98ea46ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x98ea46ce (stale) Mirror 4: 0x98ea46ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped All operations completed A-OK! PASS 70a (96s) == sanity-flr test 71: check mirror extend parent fid ==== 10:15:02 (1787580902) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x1d8:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 10:15:13 (1787580913) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x332:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257162 s, 8.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x330:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x332:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208803 s, 5.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x330:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x333:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x332:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x32f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x330:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x333:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x332:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x331:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 10:15:22 (1787580922) Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6028000 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3058016 .. failed Extending file size to 8174240 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6594272 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7771648 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8186976 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8982880 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8959104 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61564 61565 61568 61569 61571 done 192.168.201.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre2 (opts:) 192.168.201.121@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:16:42 (1787581002) Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1103232 .. Extending file size to 2476672 .. done Extending file size to 7743200 .. Extending file size to 5585152 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5872512 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9182400 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8809600 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8848256 .. Extending file size to 9334336 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63182 63183 63184 63186 63188 done 192.168.201.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre2 (opts:) 192.168.201.121@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:18:01 (1787581081) Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1268828 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25020 3582000 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 19040 3587980 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44060 7169980 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1268828 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25020 3582000 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 19040 3587980 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44060 7169980 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.268149 s, 7.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.303003 s, 6.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6f7db1a0 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:18:26 (1787581106) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0801604 s, 26.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 10:18:33 (1787581113) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e7:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x349:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x349:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34a:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x346:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:18:41 (1787581121) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146481 s, 14.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:18:50 (1787581130) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100376 s, 20.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x347:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x347:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x348:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:19:00 (1787581140) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126729 s, 16.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x58:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:19:34 (1787581174) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg121-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135753 s, 7.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x59:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x5a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x59:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x58:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:20:27 (1787581227) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg121-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x349:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x34a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160265 s, 13.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x349:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x34a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x349:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x34a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:21:22 (1787581282) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg121-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg121-server: Pool lustre.test_204f created Adding targets to pool oleg121-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg121-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147325 s, 14.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x34b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x34b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg121-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg121-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg121-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:22:35 (1787581355) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:22:42 (1787581362) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:22:51 (1787581371) oleg121-server: Pool lustre.test_206 created Adding targets to pool oleg121-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg121-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg121-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg121-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg121-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:23:23 (1787581403) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086412 s, 12.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434067 s, 24.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x7c580460 Mirror 3: 0x7c580460 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:23:37 (1787581417) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:23:41 (1787581421) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (7s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:23:48 (1787581428) Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0773706 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2500000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xc40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xc40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xc40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1b90000 Read this (4 bytes): BBBB 192.168.201.121@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg121-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:24:33 (1787581473) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0194277 s, 422 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:24:40 (1787581480) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559369 s, 18.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x206 lmm_fid: [0x200000405:0x206:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x206 lmm_fid: [0x200000405:0x206:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x359:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611654 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 10:24:41.000000000 -0400 Modify: 2026-08-24 10:24:41.000000000 -0400 Change: 2026-08-24 10:24:43.000000000 -0400 Birth: 2026-08-24 10:24:41.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:24:48 (1787581488) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400577 s, 26.2 MB/s oleg121-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg121-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg121-server: O/280000401/d21/355: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:25:12 (1787581512) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.220944 s, 185 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0453758 s, 90.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 10:25 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 10:25 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:25:20 (1787581520) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:25:28 (1787581528) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1889 sec =========== 10:25:36 (1787581536) === sanity-flr: start cleanup 10:25:38 (1787581538) === === sanity-flr: finish cleanup 10:25:46 (1787581546) ===