-----============= acceptance-small: sanity-flr ============----- Wed Aug 19 06:27:37 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg420-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 06:27:58 (1787135278) === oleg420-client.virtnet: executing check_config_client /mnt/lustre oleg420-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg420-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9f279811d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9f279811d000.idle_timeout=debug disable quota as required oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 06:28:39 (1787135319) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8228] == sanity-flr test 0a: lfs mirror create with -N option == 06:28:41 (1787135321) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:29:00 (1787135340) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:29:04 (1787135344) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:29:07 (1787135347) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:29:37 (1787135377) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:29:41 (1787135381) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 06:29:45 (1787135385) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9f279811d000.stats=clear osc.lustre-OST0001-osc-ffff9f279811d000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:29:55 (1787135395) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:30:09 (1787135409) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:30:18 (1787135418) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:30:22 (1787135422) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:30:31 (1787135431) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 06:30:41 (1787135441) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:30:49 (1787135449) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:30:58 (1787135458) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (11s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:31:09 (1787135469) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:31:24 (1787135484) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:31:33 (1787135493) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.16854 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116136 s, 9.0 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:31:45 (1787135505) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.118001 s, 35.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:31:53 (1787135513) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:32:04 (1787135524) Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f279811d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f279811d000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg420-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f279811d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f279811d000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (145s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:34:30 (1787135670) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.2169 s, 29.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f279811d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f279811d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:35:11 (1787135711) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f279811d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f279811d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f279811d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f279811d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:36:33 (1787135793) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.85891 s, 18.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 19 06:36 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.882s user 0m0.001s sys 0m0.196s Stopping /mnt/lustre-ost1 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9f279811d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9f279811d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.982s user 0m0.001s sys 0m0.201s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg420-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg420-server oleg420-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9f279811d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f279811d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.104s user 0m0.001s sys 0m0.208s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg420-server: oleg420-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg420-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f279811d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9f279811d000.ost_server_uuid in FULL state after 0 sec PASS 33b (88s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:38:02 (1787135882) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:38:05 (1787135885) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:38:09 (1787135889) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:38:13 (1787135893) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 06:38:20 (1787135900) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.442559 s, 28.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:38:32 (1787135912) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049015 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039598 s, 26.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:38:40 (1787135920) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:38:48 (1787135928) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00101192 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00126355 s, 810 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00560847 s, 183 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00198842 s, 515 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 06:38:57 (1787135937) create test files with size 15750755 .. 2b39e4deb2542b08009e7e22416548b7 - dfab8f3ed20d85805086a91772e12bc5 - 898c18595f126dc82fdfd25628ccf1c2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 06:39:18 (1787135958) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:39:32 (1787135972) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:39:39 (1787135979) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792203 s, 13.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439861 s, 23.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 06:39:49 (1787135989) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0642986 s, 32.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.470898 s, 8.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170985 s, 12.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (16s) == sanity-flr test 42: lfs mirror verify ================= 06:40:05 (1787136005) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 06:40:09 (1787136009) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:40:13 (1787136013) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239681 s, 43.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 06:40:23 (1787136023) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.409075 s, 7.7 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:40:35 (1787136035) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:40:46 (1787136046) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.241882 s, 43.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:40:59 (1787136059) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9f2784913800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.442713 s, 23.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg420-client.virtnet /mnt/lustre (opts:) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9f2785ef1800.xattr_cache=1 PASS 44e (21s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:41:20 (1787136080) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:42:04 (1787136124) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:42:46 (1787136166) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:42:50 (1787136170) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.324889 s, 9.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.144888 s, 21.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc09420e4 (nosync,stale) Mirror 2: 0xc09420e4 (nosync,stale) ** verify mirror 2 doesn't change original checksum: c3a16082c30df030b4915fb949919a16 - mirror 1 checksum: 384c5a4ab18a73f378095abc98588562 - mirror 2 checksum: c3a16082c30df030b4915fb949919a16 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T06:42:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T06:42:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c3a16082c30df030b4915fb949919a16 - mirror 1 checksum: 384c5a4ab18a73f378095abc98588562 - mirror 2 checksum: 384c5a4ab18a73f378095abc98588562 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T06:42:53-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T06:42:53-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 06:43:02 (1787136182) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488717 s, 21.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:43:10 (1787136190) Failing mds1 on oleg420-server Stopping /mnt/lustre-mds1 (opts:) on oleg420-server 06:43:22 (1787136202) shut down facet: mds1 facet_host: oleg420-server facet_failover_host: oleg420-server Failover mds1 to oleg420-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 06:43:47 (1787136227) targets are mounted 06:43:47 (1787136227) facet_failover done oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517633 s, 20.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:44:04 (1787136244) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0636903 s, 322 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0231358 s, 885 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:44:15 (1787136255) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0563389 s, 364 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0211894 s, 967 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:44:27 (1787136267) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00741641 s, 2.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:44:36 (1787136276) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0757015 s, 271 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:44:46 (1787136286) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00557331 s, 735 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0261067 s, 157 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:44:54 (1787136294) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:45:19 (1787136319) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:45:50 (1787136350) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:46:15 (1787136375) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:46:39 (1787136399) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 06:46:47 (1787136407) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57592 mirror split pid 57593 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57592 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57593 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 06:47:56 (1787136476) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 6479 fallocating to largest ever: 0x40000 000001 1787136479.438904 fallocate 0x036ae8 thru 0x03ffff (0x9518 bytes) 000002 1787136479.450594 trunc from 0x040000 to 0x022ac0 (0x1d53f bytes) 000003 1787136479.481125 trunc from 0x022ac1 to 0x0172b6 (0xb80a bytes) 000004 1787136479.504359 fallocate 0x00f82b thru 0x01e7ca (0xefa0 bytes) 000005 1787136479.510023 fallocate 0x011222 thru 0x01944c (0x822b bytes) 000006 1787136479.515982 read 0x00f7d9 thru 0x019c6c (0xa494 bytes) 000007 1787136479.532972 fallocate 0x03c712 thru 0x03ffff (0x38ee bytes) 000008 1787136479.543139 trunc from 0x01e7cb to 0x03fad3 (0x21309 bytes) 000009 1787136479.564080 MIRROR_EXTEND to 2 mirrors 000010 1787136480.341950 write 0x0018b1 thru 0x005733 (0x3e83 bytes) 000011 1787136480.434388 mapwrite 0x01451d thru 0x014542 (0x26 bytes) 000012 1787136480.478881 mapwrite 0x0272e8 thru 0x02cf96 (0x5caf bytes) 000014 1787136480.578619 trunc from 0x03fad4 to 0x000e0d (0x3ecc6 bytes) 000015 1787136480.658381 punch from 0x0001de to 0x000e0d (0xc30 bytes) 000016 1787136480.672356 punch from 0x00062e to 0x000e0d (0x7e0 bytes) 000017 1787136480.682750 trunc from 0x000e0e to 0x02f13b (0x2e32e bytes) 000018 1787136480.721565 trunc from 0x01cd2d to 0x029c5d (0xcf31 bytes) 000019 1787136480.731815 mapwrite 0x03d2a4 thru 0x03ffff (0x2d5c bytes) 000020 1787136480.831368 mapwrite 0x005001 thru 0x007970 (0x2970 bytes) 000021 1787136480.882760 read 0x011148 thru 0x017d61 (0x6c1a bytes) 000022 1787136480.898484 trunc from 0x040000 to 0x01ce9e (0x23161 bytes) 000024 1787136480.922964 punch from 0x0050f9 to 0x011111 (0xc019 bytes) 000025 1787136480.930790 trunc from 0x01ce9f to 0x01838d (0x4b11 bytes) 000026 1787136480.956735 mapwrite 0x0071f3 thru 0x011e4a (0xac58 bytes) 000027 1787136481.035698 fallocate 0x01df5b thru 0x02b4a0 (0xd546 bytes) 000028 1787136481.048730 trunc from 0x02b4a1 to 0x00e2ac (0x1d1f4 bytes) 000029 1787136481.097477 mapwrite 0x009015 thru 0x00b8bc (0x28a8 bytes) 000030 1787136481.130784 mapwrite 0x03f5b5 thru 0x03ffff (0xa4b bytes) 000031 1787136481.198500 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x11eb4d06 Mirror 3: 0x11eb4d06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000032 1787136482.389162 write 0x006905 thru 0x00eb4a (0x8246 bytes) 000033 1787136482.466963 mapread 0x027580 thru 0x03722f (0xfcb0 bytes) 000035 1787136482.526120 write 0x02f330 thru 0x036193 (0x6e64 bytes) 000036 1787136482.535257 mapwrite 0x00a860 thru 0x0115cd (0x6d6e bytes) 000038 1787136482.605855 write 0x00a267 thru 0x010c89 (0x6a23 bytes) 000039 1787136482.612964 trunc from 0x010102 to 0x0133ac (0x32ab bytes) 000040 1787136482.677323 write 0x0313cb thru 0x03b270 (0x9ea6 bytes) 000041 1787136482.690702 mapwrite 0x026a0c thru 0x03475c (0xdd51 bytes) 000042 1787136482.744728 read 0x00cc7e thru 0x010d2b (0x40ae bytes) 000043 1787136482.767041 mapread 0x011c04 thru 0x015cb1 (0x40ae bytes) 000044 1787136482.790981 mapread 0x00c92a thru 0x018e2a (0xc501 bytes) 000045 1787136482.797797 write 0x002ea8 thru 0x010ff0 (0xe149 bytes) 000046 1787136482.825936 mapread 0x038b4e thru 0x03ffff (0x74b2 bytes) 000047 1787136482.849724 punch from 0x01cfe7 to 0x02a662 (0xd67c bytes) 000048 1787136482.864676 punch from 0x002457 to 0x0074ce (0x5078 bytes) 000049 1787136482.895400 fallocate 0x012ed4 thru 0x0198db (0x6a08 bytes) 000050 1787136482.914927 trunc from 0x040000 to 0x0001a9 (0x3fe56 bytes) 000051 1787136483.019787 fallocate 0x007e9d thru 0x0157b2 (0xd916 bytes) 000053 1787136483.033316 trunc from 0x0001aa to 0x019a41 (0x19898 bytes) 000054 1787136483.050970 write 0x0382e0 thru 0x039e99 (0x1bba bytes) 000055 1787136483.057678 mapread 0x0202d5 thru 0x026a49 (0x6775 bytes) 000056 1787136483.077716 mapread 0x014aa0 thru 0x015c46 (0x11a7 bytes) 000057 1787136483.115709 punch from 0x000cc6 to 0x00427c (0x35b7 bytes) 000058 1787136483.121377 trunc from 0x039e9a to 0x031185 (0x8d14 bytes) 000061 1787136483.155152 punch from 0x01c38c to 0x01d793 (0x1408 bytes) 000062 1787136483.164135 mapwrite 0x02c833 thru 0x03868a (0xbe58 bytes) 000063 1787136483.230018 mapwrite 0x00b484 thru 0x00c4e8 (0x1065 bytes) 000064 1787136483.254370 punch from 0x01202f to 0x0130b6 (0x1088 bytes) 000065 1787136483.260580 fallocate 0x03ed13 thru 0x03ffff (0x12ed bytes) 000066 1787136483.266161 trunc from 0x040000 to 0x0235b8 (0x1ca47 bytes) 000069 1787136483.307342 mapread 0x01fce8 thru 0x0235b8 (0x38d1 bytes) 000070 1787136483.317693 trunc from 0x0235b9 to 0x02ccd1 (0x9719 bytes) 000071 1787136483.340068 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ccd2): Mirror 2: 0xe43e192a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ccd2): Mirror 2: 0xe43e192a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ccd2: skipped 000072 1787136483.682271 write 0x01fe54 thru 0x022aa6 (0x2c53 bytes) 000073 1787136483.698684 fallocate 0x021de5 thru 0x0271ed (0x5409 bytes) 000074 1787136483.704304 mapread 0x00642a thru 0x00f3e7 (0x8fbe bytes) 000075 1787136483.734896 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) 000076 1787136483.891049 mapread 0x01d073 thru 0x025a70 (0x89fe bytes) 000077 1787136483.913892 trunc from 0x018648 to 0x01aa48 (0x2401 bytes) 000078 1787136483.919328 punch from 0x01256d to 0x01d97f (0xb413 bytes) 000081 1787136483.928987 mapwrite 0x02aa4b thru 0x03a354 (0xf90a bytes) 000082 1787136484.059238 fallocate 0x03abab thru 0x03ffff (0x5455 bytes) 000083 1787136484.066500 mapwrite 0x02b7c4 thru 0x038ffb (0xd838 bytes) 000084 1787136484.098777 mapwrite 0x011f57 thru 0x01927e (0x7328 bytes) 000087 1787136484.180389 mapread 0x034a4e thru 0x0389fa (0x3fad bytes) 000089 1787136484.185741 punch from 0x0160fa to 0x0229ef (0xc8f6 bytes) 000090 1787136484.201752 write 0x014f54 thru 0x016f30 (0x1fdd bytes) 000091 1787136484.209489 mapwrite 0x018144 thru 0x01f26a (0x7127 bytes) 000092 1787136484.249642 write 0x03c2a5 thru 0x03ffff (0x3d5b bytes) 000093 1787136484.251260 trunc from 0x00e2ae to 0x013845 (0x5598 bytes) 000094 1787136484.258743 trunc from 0x040000 to 0x02fc00 (0x103ff bytes) 000095 1787136484.280452 write 0x006636 thru 0x013bb5 (0xd580 bytes) 000097 1787136484.296677 write 0x029fcb thru 0x030fad (0x6fe3 bytes) 000098 1787136484.298810 mapread 0x02e061 thru 0x030fad (0x2f4d bytes) 000099 1787136484.301128 mapread 0x02f84a thru 0x030fad (0x1764 bytes) 000101 1787136484.305558 trunc from 0x030fae to 0x03ccf8 (0xbd4b bytes) 000102 1787136484.361855 trunc from 0x01a746 to 0x0208b7 (0x6172 bytes) 000103 1787136484.379978 fallocate 0x015c02 thru 0x01d722 (0x7b21 bytes) 000105 1787136484.394475 punch from 0x02641b to 0x02aecb (0x4ab1 bytes) 000106 1787136484.449646 write 0x022360 thru 0x02c3fb (0xa09c bytes) 000107 1787136484.470217 fallocate 0x0228ba thru 0x02e3fc (0xbb43 bytes) 000108 1787136484.504271 write 0x013b56 thru 0x022db1 (0xf25c bytes) 000109 1787136484.522315 trunc from 0x014004 to 0x01ffdf (0xbfdc bytes) 000110 1787136484.603780 write 0x03398f thru 0x03ffff (0xc671 bytes) 000111 1787136484.635751 trunc from 0x026ceb to 0x02a8f0 (0x3c06 bytes) 000112 1787136484.652397 fallocate 0x033590 thru 0x03da2e (0xa49f bytes) 000114 1787136484.672872 trunc from 0x040000 to 0x00b580 (0x34a7f bytes) 000115 1787136484.721548 punch from 0x00afef to 0x00b580 (0x592 bytes) 000116 1787136484.739037 punch from 0x00a6ae to 0x00b580 (0xed3 bytes) 000117 1787136484.770496 trunc from 0x00b581 to 0x00dd18 (0x2798 bytes) 000118 1787136484.823227 mapwrite 0x00c85b thru 0x01c664 (0xfe0a bytes) 000119 1787136484.986221 mapread 0x01bc69 thru 0x01c664 (0x9fc bytes) 000120 1787136484.988661 trunc from 0x001388 to 0x00546e (0x40e7 bytes) 000121 1787136485.008318 punch from 0x01c4fb to 0x01c664 (0x16a bytes) 000122 1787136485.019905 trunc from 0x01c665 to 0x0395e5 (0x1cf81 bytes) 000123 1787136485.041524 fallocate 0x03565c thru 0x03b786 (0x612b bytes) 000124 1787136485.049273 trunc from 0x0395e6 to 0x00520b (0x343da bytes) 000125 1787136485.075384 mapread 0x003226 thru 0x00520b (0x1fe6 bytes) 000126 1787136485.091898 mapread 0x002080 thru 0x00520b (0x318c bytes) 000127 1787136485.092966 write 0x032d41 thru 0x03ef02 (0xc1c2 bytes) 000128 1787136485.111466 mapread 0x03a84d thru 0x03ef02 (0x46b6 bytes) 000129 1787136485.114595 trunc from 0x03ef03 to 0x03e6bb (0x847 bytes) 000130 1787136485.132281 mapread 0x012629 thru 0x0186e4 (0x60bc bytes) 000132 1787136485.146285 punch from 0x02ac4c to 0x0301d3 (0x5588 bytes) 000133 1787136485.155619 fallocate 0x02921c thru 0x02b180 (0x1f65 bytes) 000134 1787136485.163770 trunc from 0x03e6bc to 0x0056ab (0x39010 bytes) 000135 1787136485.183644 trunc from 0x0056ac to 0x021a97 (0x1c3ec bytes) 000137 1787136485.198275 trunc from 0x003e95 to 0x004117 (0x283 bytes) 000138 1787136485.202661 trunc from 0x021a98 to 0x00ce59 (0x14c3e bytes) 000139 1787136485.216743 mapwrite 0x00c90b thru 0x00cca7 (0x39d bytes) 000140 1787136485.248535 fallocate 0x024e40 thru 0x02fb6c (0xad2d bytes) 000141 1787136485.258354 trunc from 0x02fb6d to 0x00abba (0x24fb2 bytes) 000142 1787136485.299917 write 0x01ccae thru 0x02a953 (0xdca6 bytes) 000144 1787136485.305955 fallocate 0x01c257 thru 0x026d33 (0xaadd bytes) 000145 1787136485.311669 mapread 0x00f682 thru 0x01c3c6 (0xcd45 bytes) 000146 1787136485.330934 read 0x00f1ef thru 0x01a701 (0xb513 bytes) 000148 1787136485.333543 mapwrite 0x02a71b thru 0x02cc95 (0x257b bytes) 000149 1787136485.378670 trunc from 0x02cc96 to 0x02c123 (0xb72 bytes) 000151 1787136485.403965 punch from 0x0071b7 to 0x00b2d3 (0x411d bytes) 000152 1787136485.410516 mapread 0x0175c1 thru 0x021376 (0x9db6 bytes) 000153 1787136485.412493 read 0x01761c thru 0x01f3c8 (0x7dad bytes) 000154 1787136485.414971 fallocate 0x012dca thru 0x01eb44 (0xbd7b bytes) 000155 1787136485.421581 read 0x00c187 thru 0x014e6c (0x8ce6 bytes) 000156 1787136485.422516 mapread 0x008a22 thru 0x00eaf9 (0x60d8 bytes) 000158 1787136485.436019 fallocate 0x031668 thru 0x0326c6 (0x105f bytes) 000160 1787136485.443337 read 0x00dce4 thru 0x01ac00 (0xcf1d bytes) 000161 1787136485.444614 mapwrite 0x03ce63 thru 0x03ffff (0x319d bytes) 000162 1787136485.496849 trunc from 0x040000 to 0x00da8e (0x32571 bytes) 000163 1787136485.524467 read 0x00c1a7 thru 0x00da8e (0x18e8 bytes) 000164 1787136485.526519 mapwrite 0x012120 thru 0x0176ba (0x559b bytes) 000165 1787136485.601654 trunc from 0x0176bb to 0x0242aa (0xcbf0 bytes) 000166 1787136485.623768 write 0x00e653 thru 0x0182e7 (0x9c95 bytes) 000167 1787136485.638284 read 0x001e9c thru 0x00c02f (0xa194 bytes) 000168 1787136485.639291 trunc from 0x01154f to 0x01d3b7 (0xbe69 bytes) 000170 1787136485.683218 trunc from 0x0242ab to 0x019d6b (0xa53f bytes) 000171 1787136485.704206 trunc from 0x019d6c to 0x036f2d (0x1d1c2 bytes) 000172 1787136485.738204 trunc from 0x036f2e to 0x01601a (0x20f13 bytes) 000173 1787136485.768467 write 0x037956 thru 0x03b3f9 (0x3aa4 bytes) 000174 1787136485.769743 write 0x013333 thru 0x02133c (0xe00a bytes) 000175 1787136485.785669 trunc from 0x03b3fa to 0x01393c (0x27abd bytes) 000177 1787136485.827649 fallocate 0x0289ab thru 0x0363cd (0xda23 bytes) 000178 1787136485.838964 write 0x03e1d3 thru 0x03ffff (0x1e2d bytes) 000179 1787136485.840565 read 0x00813b thru 0x00ebce (0x6a94 bytes) 000180 1787136485.843815 mapread 0x005032 thru 0x013fc8 (0xef97 bytes) 000181 1787136485.849188 write 0x00f87f thru 0x01801c (0x879e bytes) 000183 1787136485.870725 trunc from 0x040000 to 0x00df5c (0x320a3 bytes) 000184 1787136485.907222 trunc from 0x00df5d to 0x02ad86 (0x1ce2a bytes) 000185 1787136485.927154 read 0x02435d thru 0x02ad86 (0x6a2a bytes) 000188 1787136485.960441 trunc from 0x02963b to 0x02ad86 (0x174c bytes) 000189 1787136485.970728 mapread 0x0037ae thru 0x008417 (0x4c6a bytes) 000190 1787136485.972744 trunc from 0x02ad87 to 0x019b5a (0x1122c bytes) 000191 1787136485.998739 trunc from 0x014bc2 to 0x019b5a (0x4f99 bytes) 000192 1787136486.009908 mapwrite 0x0271b9 thru 0x031758 (0xa5a0 bytes) 000193 1787136486.101285 write 0x0236c9 thru 0x02691e (0x3256 bytes) 000194 1787136486.133541 MIRROR_EXTEND to 2 mirrors 000195 1787136487.123410 punch from 0x0181c0 to 0x018500 (0x341 bytes) 000197 1787136487.193694 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) 000198 1787136487.350674 MIRROR_EXTEND to 2 mirrors 000199 1787136488.135635 trunc from 0x00f18c to 0x0178f1 (0x8766 bytes) 000200 1787136488.205681 MIRROR_RESYNC 2 mirrors 000201 1787136488.864532 mapread 0x0055ef thru 0x0137f6 (0xe208 bytes) 000202 1787136488.894027 punch from 0x0089ef to 0x017c23 (0xf235 bytes) 000204 1787136488.939593 trunc from 0x001ca3 to 0x01107b (0xf3d9 bytes) 000206 1787136488.947036 write 0x023c01 thru 0x030f4f (0xd34f bytes) 000207 1787136488.959683 mapwrite 0x001760 thru 0x00a35d (0x8bfe bytes) 000208 1787136488.992302 trunc from 0x0037f0 to 0x00ba20 (0x8231 bytes) 000209 1787136489.000656 mapread 0x007e2c thru 0x015567 (0xd73c bytes) 000210 1787136489.046471 trunc from 0x031759 to 0x03db88 (0xc430 bytes) 000211 1787136489.083604 write 0x02629c thru 0x032b2b (0xc890 bytes) 000212 1787136489.104263 read 0x017261 thru 0x021fce (0xad6e bytes) 000214 1787136489.134611 fallocate 0x03989b thru 0x03a7a5 (0xf0b bytes) 000215 1787136489.145533 mapread 0x02cffc thru 0x02fb2c (0x2b31 bytes) 000216 1787136489.152646 trunc from 0x036082 to 0x03db88 (0x7b07 bytes) 000217 1787136489.198520 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) 000219 1787136489.471121 mapread 0x02dade thru 0x033682 (0x5ba5 bytes) 000220 1787136489.526068 punch from 0x033039 to 0x03db88 (0xab50 bytes) 000221 1787136489.540227 write 0x035a9d thru 0x036c47 (0x11ab bytes) 000222 1787136489.576761 punch from 0x03d6a8 to 0x03db88 (0x4e1 bytes) 000223 1787136489.585250 trunc from 0x01ccce to 0x0255e9 (0x891c bytes) 000224 1787136489.608399 read 0x0157c8 thru 0x01edb1 (0x95ea bytes) 000225 1787136489.671184 write 0x03c266 thru 0x03ffff (0x3d9a bytes) 000227 1787136489.685767 trunc from 0x029644 to 0x029f63 (0x920 bytes) 000228 1787136489.699459 mapread 0x02f0cc thru 0x03ec61 (0xfb96 bytes) 000229 1787136489.734235 MIRROR_EXTEND to 2 mirrors 000230 1787136490.975178 read 0x009c5e thru 0x019bdd (0xff80 bytes) 000231 1787136490.988890 fallocate 0x00b8a9 thru 0x00e61e (0x2d76 bytes) 000232 1787136491.029322 fallocate 0x031c95 thru 0x037481 (0x57ed bytes) 000233 1787136491.036156 punch from 0x018d6d to 0x0244f5 (0xb789 bytes) 000234 1787136491.051686 fallocate 0x0093a3 thru 0x0175fb (0xe259 bytes) 000235 1787136491.062916 punch from 0x02c2b7 to 0x03918a (0xced4 bytes) 000236 1787136491.071086 trunc from 0x009bd8 to 0x013c83 (0xa0ac bytes) 000237 1787136491.077548 punch from 0x0198a0 to 0x024fd6 (0xb737 bytes) 000238 1787136491.087569 read 0x01f0aa thru 0x02d06f (0xdfc6 bytes) 000239 1787136491.121633 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) 000240 1787136491.365598 trunc from 0x02f36d to 0x03cf1e (0xdbb2 bytes) 000241 1787136491.377789 trunc from 0x007384 to 0x008a32 (0x16af bytes) 000242 1787136491.384632 trunc from 0x0298fa to 0x03857c (0xec83 bytes) 000243 1787136491.392970 fallocate 0x01b8ad thru 0x027bce (0xc322 bytes) 000244 1787136491.400562 mapwrite 0x009d26 thru 0x014e3f (0xb11a bytes) 000245 1787136491.481105 trunc from 0x00f00d to 0x0165bf (0x75b3 bytes) 000246 1787136491.495596 fallocate 0x00f1db thru 0x019525 (0xa34b bytes) 000247 1787136491.507309 trunc from 0x0051b9 to 0x008e5d (0x3ca5 bytes) 000248 1787136491.514539 mapwrite 0x00d534 thru 0x0132ae (0x5d7b bytes) 000249 1787136491.557900 read 0x0001e7 thru 0x007a92 (0x78ac bytes) 000250 1787136491.576500 punch from 0x03d528 to 0x03ffff (0x2ad8 bytes) 000251 1787136491.588966 write 0x014772 thru 0x01ab9a (0x6429 bytes) 000252 1787136491.605785 trunc from 0x03ccbe to 0x03ffff (0x3342 bytes) 000253 1787136491.617724 fallocate 0x0287a2 thru 0x032de5 (0xa644 bytes) 000254 1787136491.627321 mapread 0x02be06 thru 0x03add5 (0xefd0 bytes) 000255 1787136491.659498 write 0x0076ab thru 0x0078d5 (0x22b bytes) 000256 1787136491.667217 trunc from 0x020ade to 0x02183c (0xd5f bytes) 000257 1787136491.676696 mapwrite 0x003b08 thru 0x00b731 (0x7c2a bytes) 000258 1787136491.730892 trunc from 0x00afca to 0x00b2de (0x315 bytes) 000259 1787136491.752890 punch from 0x028634 to 0x030773 (0x8140 bytes) 000260 1787136491.768529 read 0x000e7a thru 0x00f89a (0xea21 bytes) 000261 1787136491.797460 mapread 0x039a5f thru 0x03bc25 (0x21c7 bytes) 000262 1787136491.802019 punch from 0x00ed14 to 0x016b2f (0x7e1c bytes) 000263 1787136491.814489 mapread 0x014aa7 thru 0x01c17f (0x76d9 bytes) 000265 1787136491.867699 write 0x03a502 thru 0x03ffff (0x5afe bytes) 000267 1787136491.872503 trunc from 0x02d789 to 0x0368f5 (0x916d bytes) 000268 1787136491.881687 trunc from 0x031d33 to 0x03cfe6 (0xb2b4 bytes) 000271 1787136491.920622 mapread 0x0025f7 thru 0x002796 (0x1a0 bytes) 000272 1787136491.922005 mapread 0x01cba4 thru 0x027361 (0xa7be bytes) 000273 1787136491.926243 fallocate 0x0094b2 thru 0x01427a (0xadc9 bytes) 000274 1787136491.940811 fallocate 0x024690 thru 0x02b955 (0x72c6 bytes) 000275 1787136491.961929 trunc from 0x040000 to 0x032328 (0xdcd7 bytes) 000276 1787136491.990445 fallocate 0x013e4e thru 0x01d6b4 (0x9867 bytes) 000277 1787136491.998639 write 0x002726 thru 0x007c91 (0x556c bytes) 000278 1787136492.008752 mapread 0x016312 thru 0x02031f (0xa00e bytes) 000279 1787136492.013643 mapwrite 0x0287f5 thru 0x03306b (0xa877 bytes) 000280 1787136492.236144 punch from 0x01f570 to 0x022788 (0x3219 bytes) 000281 1787136492.246575 write 0x032963 thru 0x0356d8 (0x2d76 bytes) 000282 1787136492.252790 read 0x01fec9 thru 0x02d16c (0xd2a4 bytes) 000283 1787136492.277422 trunc from 0x0356d9 to 0x03bcbc (0x65e4 bytes) 000284 1787136492.320563 trunc from 0x028e0a to 0x02c733 (0x392a bytes) 000286 1787136492.351526 mapwrite 0x00c412 thru 0x0167d7 (0xa3c6 bytes) 000287 1787136492.410191 punch from 0x03062b to 0x03a581 (0x9f57 bytes) 000288 1787136492.422415 mapwrite 0x03bfa2 thru 0x03f600 (0x365f bytes) 000289 1787136492.576150 mapwrite 0x027570 thru 0x034c01 (0xd692 bytes) 000290 1787136492.709686 fallocate 0x0064c3 thru 0x00736d (0xeab bytes) 000291 1787136492.720680 trunc from 0x03f601 to 0x0368bc (0x8d44 bytes) 000292 1787136492.786145 read 0x01c8a1 thru 0x0223be (0x5b1e bytes) 000293 1787136492.786999 trunc from 0x003527 to 0x00a771 (0x724b bytes) 000294 1787136492.797228 mapwrite 0x019474 thru 0x021d55 (0x88e2 bytes) 000295 1787136492.841868 trunc from 0x0358d0 to 0x0368bc (0xfed bytes) 000296 1787136492.858251 trunc from 0x001ae5 to 0x008d50 (0x726c bytes) 000297 1787136492.874293 read 0x02df0a thru 0x0368bc (0x89b3 bytes) 000298 1787136492.892706 read 0x031a6e thru 0x0368bc (0x4e4f bytes) 000299 1787136492.893434 read 0x019eef thru 0x028ade (0xebf0 bytes) 000300 1787136492.894090 fallocate 0x0294a7 thru 0x0328a1 (0x93fb bytes) 000301 1787136492.923440 trunc from 0x0368bd to 0x02055f (0x1635d bytes) 000302 1787136492.958017 read 0x01bf91 thru 0x02055f (0x45cf bytes) 000303 1787136492.970443 MIRROR_EXTEND to 2 mirrors 000304 1787136493.758021 fallocate 0x022f2c thru 0x02bc88 (0x8d5d bytes) 000305 1787136493.802480 punch from 0x019bc5 to 0x01a200 (0x63c bytes) 000306 1787136493.807063 trunc from 0x02bc89 to 0x02c759 (0xad1 bytes) 000307 1787136493.830361 mapread 0x01f4a4 thru 0x02c759 (0xd2b6 bytes) 000308 1787136493.876021 trunc from 0x02c75a to 0x0066cb (0x2608e bytes) 000309 1787136493.902179 read 0x001f3b thru 0x0066cb (0x4791 bytes) 000310 1787136493.916785 punch from 0x003eb0 to 0x0066cb (0x281c bytes) 000311 1787136493.923031 read 0x0040ca thru 0x0066cb (0x2602 bytes) 000312 1787136493.938296 trunc from 0x0066cc to 0x009354 (0x2c89 bytes) 000313 1787136493.956767 mapread 0x003f35 thru 0x0060e5 (0x21b1 bytes) 000314 1787136493.960754 punch from 0x003f77 to 0x0053cc (0x1456 bytes) 000315 1787136493.967856 trunc from 0x00054d to 0x0040ab (0x3b5f bytes) 000316 1787136493.983172 punch from 0x004a6b to 0x009354 (0x48ea bytes) 000317 1787136493.992683 write 0x01d39d thru 0x02c838 (0xf49c bytes) 000318 1787136493.996789 read 0x011e27 thru 0x01b094 (0x926e bytes) 000319 1787136494.041270 mapread 0x007334 thru 0x009574 (0x2241 bytes) 000320 1787136494.043650 mapwrite 0x03ddb6 thru 0x03ffff (0x224a bytes) 000321 1787136494.142223 punch from 0x031966 to 0x03d650 (0xbceb bytes) 000322 1787136494.152214 fallocate 0x01fe3c thru 0x027917 (0x7adc bytes) 000323 1787136494.161814 trunc from 0x004fe2 to 0x0149b8 (0xf9d7 bytes) 000324 1787136494.168442 trunc from 0x03633c to 0x03fb98 (0x985d bytes) 000325 1787136494.188276 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x22a5002d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7283c2e5 Mirror 3: 0x7283c2e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000326 1787136495.590221 punch from 0x0076b1 to 0x008eda (0x182a bytes) 000327 1787136495.640221 trunc from 0x040000 to 0x01aa87 (0x25578 bytes) 000328 1787136495.713657 trunc from 0x00c51b to 0x00d746 (0x122c bytes) 000329 1787136495.719953 trunc from 0x01aa88 to 0x032ab3 (0x1802c bytes) 000330 1787136495.741498 mapread 0x003cbf thru 0x00fe50 (0xc192 bytes) 000331 1787136495.775689 mapread 0x02a3f4 thru 0x032ab3 (0x86c0 bytes) 000332 1787136495.788750 read 0x014a11 thru 0x0212df (0xc8cf bytes) 000333 1787136495.812291 mapread 0x03118c thru 0x032ab3 (0x1928 bytes) 000334 1787136495.813540 mapwrite 0x0195bb thru 0x022879 (0x92bf bytes) 000335 1787136495.866904 trunc from 0x032ab4 to 0x023b5a (0xef59 bytes) 000336 1787136495.900681 read 0x018fb5 thru 0x021bb1 (0x8bfd bytes) 000337 1787136495.902802 read 0x022934 thru 0x023b5a (0x1227 bytes) 000338 1787136495.907532 trunc from 0x0123c0 to 0x018edc (0x6b1d bytes) 000339 1787136495.934893 trunc from 0x007e05 to 0x015164 (0xd360 bytes) 000340 1787136495.941860 mapwrite 0x00d91d thru 0x00f7d7 (0x1ebb bytes) 000341 1787136495.994418 trunc from 0x023b5b to 0x023ec9 (0x36f bytes) 000342 1787136496.039523 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x23eca): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23eca): Mirror 2: 0xa875ebfd (stale) Mirror 3: 0x59041c37 (stale) CRC-32 checksum value for chunk [0, 0x23eca): Mirror 1: 0xa1eb9e27 Mirror 4: 0xa1eb9e27 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23eca: skipped 000343 1787136497.464968 punch from 0x021c03 to 0x023ec9 (0x22c7 bytes) 000344 1787136497.541013 mapwrite 0x01afab thru 0x029d31 (0xed87 bytes) 000345 1787136497.712916 punch from 0x019738 to 0x021419 (0x7ce2 bytes) 000346 1787136497.735955 MIRROR_RESYNC 4 mirrors 000347 1787136499.421664 mapread 0x0212d7 thru 0x029d31 (0x8a5b bytes) 000348 1787136499.449796 punch from 0x0230bb to 0x0292c9 (0x620f bytes) 000349 1787136499.490250 mapwrite 0x0257c0 thru 0x02769b (0x1edc bytes) 000350 1787136499.518098 fallocate 0x003a3f thru 0x00595f (0x1f21 bytes) 000351 1787136499.544223 MIRROR_RESYNC 4 mirrors 000352 1787136501.335939 trunc from 0x029d32 to 0x038c13 (0xeee2 bytes) 000353 1787136501.402124 trunc from 0x038c14 to 0x00b71b (0x2d4f8 bytes) 000354 1787136501.430898 trunc from 0x00b71c to 0x006008 (0x5713 bytes) 000355 1787136501.449588 punch from 0x000896 to 0x006008 (0x5773 bytes) 000356 1787136501.458551 read 0x0012e4 thru 0x006008 (0x4d25 bytes) 000357 1787136501.468795 fallocate 0x03b9b6 thru 0x03ffff (0x464a bytes) 000358 1787136501.478652 mapwrite 0x038fe8 thru 0x03ffff (0x7018 bytes) 000359 1787136501.544897 read 0x0375c4 thru 0x039276 (0x1cb3 bytes) 000360 1787136501.548643 trunc from 0x040000 to 0x0002d6 (0x3fd29 bytes) 000361 1787136501.582181 trunc from 0x0002d7 to 0x019140 (0x18e6a bytes) 000362 1787136501.602938 mapread 0x0060c0 thru 0x012e2a (0xcd6b bytes) 000363 1787136501.625375 read 0x0180b3 thru 0x019140 (0x108e bytes) 000365 1787136501.628335 trunc from 0x00f2fe to 0x015beb (0x68ee bytes) 000366 1787136501.636571 trunc from 0x0025ab to 0x00c9e5 (0xa43b bytes) 000370 1787136501.656284 mapwrite 0x0127c1 thru 0x02194e (0xf18e bytes) 000371 1787136501.766409 write 0x00d30b thru 0x010665 (0x335b bytes) 000372 1787136501.768874 mapwrite 0x022810 thru 0x02acd9 (0x84ca bytes) 000373 1787136501.858730 trunc from 0x01e1fa to 0x023416 (0x521d bytes) 000374 1787136501.876332 trunc from 0x02acda to 0x0113d5 (0x19904 bytes) 000375 1787136501.924648 MIRROR_RESYNC 4 mirrors 000376 1787136502.933726 mapread 0x00ea20 thru 0x0113d5 (0x29b6 bytes) 000377 1787136502.962786 punch from 0x0037f6 to 0x00b94e (0x8159 bytes) 000379 1787136503.006193 mapwrite 0x0094e3 thru 0x00c13c (0x2c5a bytes) 000380 1787136503.048918 trunc from 0x00c524 to 0x00f989 (0x3466 bytes) 000383 1787136503.058551 fallocate 0x032f55 thru 0x03bf96 (0x9042 bytes) 000384 1787136503.064321 trunc from 0x01ff35 to 0x02ae77 (0xaf43 bytes) 000385 1787136503.070537 read 0x023d69 thru 0x0253ea (0x1682 bytes) 000386 1787136503.083791 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bf97): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf97): Mirror 2: 0xdee53a4d (stale) Mirror 3: 0xdee53a4d (stale) Mirror 4: 0xdee53a4d (stale) CRC-32 checksum value for chunk [0, 0x3bf97): Mirror 1: 0x17a1282e Mirror 5: 0x17a1282e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf97: skipped 000387 1787136504.450539 read 0x01962c thru 0x020534 (0x6f09 bytes) 000388 1787136504.466448 mapwrite 0x00aae0 thru 0x011f8f (0x74b0 bytes) 000389 1787136504.609147 trunc from 0x03a44c to 0x03bf96 (0x1b4b bytes) 000390 1787136504.620656 fallocate 0x00de58 thru 0x00e19c (0x345 bytes) 000391 1787136504.630640 trunc from 0x00876c to 0x014346 (0xbbdb bytes) 000392 1787136504.639110 mapwrite 0x03aea6 thru 0x03ffff (0x515a bytes) 000393 1787136504.738961 fallocate 0x00a564 thru 0x00c2fa (0x1d97 bytes) 000394 1787136504.746050 trunc from 0x040000 to 0x03bfea (0x4015 bytes) 000395 1787136504.803927 trunc from 0x02edad to 0x0308c1 (0x1b15 bytes) 000396 1787136504.813181 punch from 0x01e31e to 0x01f9a2 (0x1685 bytes) 000397 1787136504.824942 mapwrite 0x02a09c thru 0x033654 (0x95b9 bytes) 000398 1787136504.887779 mapwrite 0x02ecc1 thru 0x03d6ed (0xea2d bytes) 000399 1787136505.017161 read 0x024a42 thru 0x027544 (0x2b03 bytes) 000400 1787136505.017802 read 0x0177d8 thru 0x01a51c (0x2d45 bytes) 000401 1787136505.048239 read 0x0100ba thru 0x0160d7 (0x601e bytes) 000402 1787136505.049134 read 0x000026 thru 0x00dde5 (0xddc0 bytes) 000404 1787136505.058560 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d6ee): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d6ee): Mirror 2: 0x3a96f4b0 (stale) Mirror 3: 0x3a96f4b0 (stale) Mirror 4: 0x3a96f4b0 (stale) Mirror 5: 0xb76e510b (stale) CRC-32 checksum value for chunk [0, 0x3d6ee): Mirror 1: 0xa5da32aa Mirror 6: 0xa5da32aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d6ee: skipped 000405 1787136506.490928 punch from 0x015a8d to 0x0215af (0xbb23 bytes) 000406 1787136506.556905 read 0x02ff0a thru 0x032f06 (0x2ffd bytes) 000407 1787136506.588165 MIRROR_RESYNC 6 mirrors 000408 1787136507.984124 mapwrite 0x03bd80 thru 0x03ffff (0x4280 bytes) 000409 1787136508.058381 write 0x00d449 thru 0x015ca0 (0x8858 bytes) 000410 1787136508.069592 trunc from 0x034e65 to 0x0358cc (0xa68 bytes) 000411 1787136508.075694 read 0x020ec2 thru 0x02d069 (0xc1a8 bytes) 000412 1787136508.088377 trunc from 0x03cd83 to 0x03ffff (0x327d bytes) 000413 1787136508.095263 write 0x024e9e thru 0x027ed2 (0x3035 bytes) 000414 1787136508.097531 mapread 0x011e47 thru 0x015916 (0x3ad0 bytes) 000415 1787136508.099395 punch from 0x0212b4 to 0x0215e7 (0x334 bytes) 000416 1787136508.104553 trunc from 0x040000 to 0x02ff93 (0x1006c bytes) 000417 1787136508.128620 trunc from 0x009b52 to 0x0197eb (0xfc9a bytes) 000418 1787136508.146327 fallocate 0x02e45d thru 0x03de06 (0xf9aa bytes) 000420 1787136508.152513 fallocate 0x01f97a thru 0x022444 (0x2acb bytes) 000421 1787136508.158010 write 0x01a9af thru 0x01b426 (0xa78 bytes) 000422 1787136508.169268 mapwrite 0x033dba thru 0x03d84c (0x9a93 bytes) 000423 1787136508.215092 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3de07): Mirror 3: 0x34ddd865 (stale) Mirror 4: 0x34ddd865 (stale) Mirror 5: 0x34ddd865 (stale) Mirror 6: 0x34ddd865 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de07): Mirror 3: 0x34ddd865 (stale) Mirror 4: 0x34ddd865 (stale) Mirror 5: 0x34ddd865 (stale) Mirror 6: 0x34ddd865 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de07: skipped 000424 1787136508.613264 mapwrite 0x035527 thru 0x03a1b8 (0x4c92 bytes) 000425 1787136508.711761 trunc from 0x03de07 to 0x026c14 (0x171f2 bytes) 000426 1787136508.756659 fallocate 0x024d59 thru 0x031161 (0xc409 bytes) 000427 1787136508.767287 punch from 0x009062 to 0x00bedc (0x2e7b bytes) 000430 1787136508.786253 MIRROR_RESYNC 5 mirrors 000431 1787136510.432641 mapread 0x01c4d2 thru 0x0258e1 (0x9410 bytes) 000432 1787136510.478161 MIRROR_EXTEND to 6 mirrors 000433 1787136511.603752 mapread 0x021e91 thru 0x030d00 (0xee70 bytes) 000434 1787136511.667441 write 0x03a53b thru 0x03ffff (0x5ac5 bytes) 000435 1787136511.749106 fallocate 0x02a9c0 thru 0x02fbca (0x520b bytes) 000436 1787136511.790412 MIRROR_RESYNC 6 mirrors 000437 1787136513.915361 read 0x01d21d thru 0x027cad (0xaa91 bytes) 000438 1787136513.936116 punch from 0x015924 to 0x01cd99 (0x7476 bytes) 000439 1787136513.973115 fallocate 0x028750 thru 0x0374ab (0xed5c bytes) 000440 1787136513.981978 trunc from 0x040000 to 0x02ec4d (0x113b2 bytes) 000441 1787136514.012408 mapwrite 0x025759 thru 0x0325c3 (0xce6b bytes) 000442 1787136514.152314 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x325c4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x325c4): Mirror 3: 0x681ee91 (stale) Mirror 4: 0x681ee91 (stale) Mirror 5: 0x681ee91 (stale) Mirror 6: 0x681ee91 (stale) Mirror 7: 0x681ee91 (stale) CRC-32 checksum value for chunk [0, 0x325c4): Mirror 1: 0xec6d005e Mirror 8: 0xec6d005e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x325c4: skipped 000443 1787136515.513419 mapwrite 0x035694 thru 0x03ffff (0xa96c bytes) 000444 1787136515.808711 mapwrite 0x016186 thru 0x02087a (0xa6f5 bytes) 000445 1787136515.874125 mapwrite 0x03f687 thru 0x03ffff (0x979 bytes) 000446 1787136515.893977 write 0x02ab15 thru 0x038da0 (0xe28c bytes) 000447 1787136515.905062 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbefe6087 (stale) Mirror 4: 0xbefe6087 (stale) Mirror 5: 0xbefe6087 (stale) Mirror 6: 0xbefe6087 (stale) Mirror 7: 0xbefe6087 (stale) Mirror 8: 0x2f83becb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x36f58e32 Mirror 9: 0x36f58e32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000449 1787136517.189320 punch from 0x00516c to 0x00d396 (0x822b bytes) 000450 1787136517.260468 mapread 0x03b543 thru 0x03ffff (0x4abd bytes) 000451 1787136517.301300 write 0x03b2c0 thru 0x03ffff (0x4d40 bytes) 000452 1787136517.308527 mapwrite 0x03b82c thru 0x03ffff (0x47d4 bytes) 000453 1787136517.347402 trunc from 0x0393d7 to 0x039e8b (0xab5 bytes) 000454 1787136517.356307 read 0x0122a8 thru 0x02041b (0xe174 bytes) 000455 1787136517.372092 trunc from 0x02dd77 to 0x0326c8 (0x4952 bytes) 000456 1787136517.384590 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbefe6087 (stale) Mirror 4: 0xbefe6087 (stale) Mirror 5: 0xbefe6087 (stale) Mirror 6: 0xbefe6087 (stale) Mirror 7: 0xbefe6087 (stale) Mirror 8: 0x2f83becb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbefe6087 (stale) Mirror 4: 0xbefe6087 (stale) Mirror 5: 0xbefe6087 (stale) Mirror 6: 0xbefe6087 (stale) Mirror 7: 0xbefe6087 (stale) Mirror 8: 0x2f83becb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000457 1787136517.998159 write 0x0044f8 thru 0x011039 (0xcb42 bytes) 000458 1787136518.013900 trunc from 0x01a4d4 to 0x01d68a (0x31b7 bytes) 000459 1787136518.024664 read 0x022482 thru 0x02b56a (0x90e9 bytes) 000460 1787136518.042080 fallocate 0x024a63 thru 0x0281aa (0x3748 bytes) 000461 1787136518.047028 trunc from 0x01447c to 0x01adbd (0x6942 bytes) 000462 1787136518.057559 MIRROR_RESYNC 7 mirrors 000464 1787136521.430519 mapwrite 0x00d91c thru 0x01d456 (0xfb3b bytes) 000465 1787136521.580547 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x705ad6f (stale) Mirror 4: 0x705ad6f (stale) Mirror 5: 0x705ad6f (stale) Mirror 6: 0x705ad6f (stale) Mirror 7: 0x705ad6f (stale) Mirror 8: 0x705ad6f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9daeae0a Mirror 9: 0x9daeae0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000466 1787136522.952230 write 0x008e98 thru 0x00927c (0x3e5 bytes) 000467 1787136523.013959 mapwrite 0x02df75 thru 0x03685a (0x88e6 bytes) 000468 1787136523.059688 mapread 0x021164 thru 0x02a51c (0x93b9 bytes) 000469 1787136523.070716 trunc from 0x02682c to 0x026cbe (0x493 bytes) 000470 1787136523.075718 trunc from 0x040000 to 0x029a4e (0x165b1 bytes) 000471 1787136523.104708 mapread 0x004841 thru 0x00e6f3 (0x9eb3 bytes) 000472 1787136523.119548 MIRROR_RESYNC 8 mirrors 000473 1787136525.088019 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) 000474 1787136525.728474 trunc from 0x028fcc to 0x029a4e (0xa83 bytes) 000475 1787136525.816430 trunc from 0x029a4f to 0x0396de (0xfc90 bytes) 000476 1787136525.892848 trunc from 0x0396df to 0x031621 (0x80bd bytes) 000477 1787136525.925901 fallocate 0x0172bc thru 0x025526 (0xe26b bytes) 000478 1787136525.946641 read 0x00502e thru 0x008cd9 (0x3cac bytes) 000479 1787136525.963863 punch from 0x026e64 to 0x031621 (0xa7be bytes) 000481 1787136525.977417 read 0x006966 thru 0x00a9cf (0x406a bytes) 000482 1787136526.001060 write 0x035e3a thru 0x0397c3 (0x398a bytes) 000483 1787136526.008108 read 0x02ba5a thru 0x030594 (0x4b3b bytes) 000484 1787136526.013439 mapread 0x00af5e thru 0x014b42 (0x9be5 bytes) 000485 1787136526.020523 trunc from 0x0397c4 to 0x031dfb (0x79c8 bytes) 000486 1787136526.053757 trunc from 0x031dfc to 0x001cb1 (0x3014a bytes) 000487 1787136526.094010 trunc from 0x0006bc to 0x001cb1 (0x15f6 bytes) 000488 1787136526.103534 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cb2): Mirror 3: 0x69e5d7dc (stale) Mirror 4: 0x69e5d7dc (stale) Mirror 5: 0x69e5d7dc (stale) Mirror 7: 0x69e5d7dc (stale) Mirror 9: 0x69e5d7dc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cb2): Mirror 3: 0x69e5d7dc (stale) Mirror 4: 0x69e5d7dc (stale) Mirror 5: 0x69e5d7dc (stale) Mirror 7: 0x69e5d7dc (stale) Mirror 9: 0x69e5d7dc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cb2: skipped 000489 1787136526.706068 fallocate 0x016032 thru 0x021f38 (0xbf07 bytes) 000490 1787136526.715085 trunc from 0x001cb2 to 0x036f3a (0x35289 bytes) 000491 1787136526.749192 mapwrite 0x02c8be thru 0x038a8a (0xc1cd bytes) 000492 1787136526.872925 mapwrite 0x013177 thru 0x01e5b2 (0xb43c bytes) 000493 1787136526.981789 mapread 0x01a62c thru 0x02a338 (0xfd0d bytes) 000494 1787136526.996793 write 0x004525 thru 0x00be49 (0x7925 bytes) 000495 1787136527.026135 trunc from 0x038a8b to 0x0137c8 (0x252c2 bytes) 000496 1787136527.055791 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x137c9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x137c9): Mirror 3: 0x5bc098f3 (stale) Mirror 4: 0x5bc098f3 (stale) Mirror 5: 0x5bc098f3 (stale) Mirror 7: 0x5bc098f3 (stale) Mirror 9: 0x5bc098f3 (stale) CRC-32 checksum value for chunk [0, 0x137c9): Mirror 1: 0xa798842a Mirror 10: 0xa798842a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x137c9: skipped 000497 1787136528.495940 trunc from 0x00fafa to 0x0137c8 (0x3ccf bytes) 000499 1787136528.555086 MIRROR_RESYNC 7 mirrors 000500 1787136530.828432 trunc from 0x005fa4 to 0x006707 (0x764 bytes) 000501 1787136530.906739 mapwrite 0x00f077 thru 0x014852 (0x57dc bytes) 000502 1787136531.009687 fallocate 0x00cb3d thru 0x014df8 (0x82bc bytes) 000503 1787136531.024978 trunc from 0x014df9 to 0x02aac9 (0x15cd1 bytes) 000504 1787136531.066733 trunc from 0x00261a to 0x00b812 (0x91f9 bytes) 000506 1787136531.075471 mapwrite 0x0068e0 thru 0x00d76c (0x6e8d bytes) 000508 1787136531.116043 mapread 0x009470 thru 0x009a1b (0x5ac bytes) 000509 1787136531.119676 trunc from 0x01276a to 0x01e063 (0xb8fa bytes) 000510 1787136531.129047 fallocate 0x02a601 thru 0x02fc52 (0x5652 bytes) 000511 1787136531.139583 trunc from 0x00ef91 to 0x01d26f (0xe2df bytes) 000512 1787136531.154591 read 0x001a8e thru 0x00a03b (0x85ae bytes) 000513 1787136531.155373 fallocate 0x0149f0 thru 0x01e9cb (0x9fdc bytes) 000514 1787136531.167180 punch from 0x02aa05 to 0x02aac9 (0xc5 bytes) 000515 1787136531.176462 fallocate 0x0326df thru 0x03562c (0x2f4e bytes) 000516 1787136531.181755 mapwrite 0x013642 thru 0x018844 (0x5203 bytes) 000517 1787136531.260002 mapread 0x003a12 thru 0x010702 (0xccf1 bytes) 000518 1787136531.263992 write 0x01c3fd thru 0x026a50 (0xa654 bytes) 000519 1787136531.279881 mapread 0x026e83 thru 0x034762 (0xd8e0 bytes) 000520 1787136531.308586 read 0x016e46 thru 0x0269f8 (0xfbb3 bytes) 000521 1787136531.310081 read 0x02abc3 thru 0x0328ca (0x7d08 bytes) 000522 1787136531.310956 mapread 0x03080f thru 0x033ecc (0x36be bytes) 000524 1787136531.313350 mapread 0x021600 thru 0x024179 (0x2b7a bytes) 000525 1787136531.315287 read 0x031c9a thru 0x03562c (0x3993 bytes) 000526 1787136531.316545 trunc from 0x002c72 to 0x0085af (0x593e bytes) 000528 1787136531.327737 trunc from 0x00c36d to 0x014b45 (0x87d9 bytes) 000529 1787136531.335350 fallocate 0x01dd86 thru 0x01f533 (0x17ae bytes) 000530 1787136531.341062 trunc from 0x00f47f to 0x0128f4 (0x3476 bytes) 000531 1787136531.347348 write 0x032bf3 thru 0x03c44d (0x985b bytes) 000532 1787136531.353174 write 0x03488f thru 0x03e362 (0x9ad4 bytes) 000533 1787136531.354779 mapwrite 0x020e8c thru 0x02c6f6 (0xb86b bytes) 000534 1787136531.415262 write 0x021db6 thru 0x024cf1 (0x2f3c bytes) 000536 1787136531.418289 trunc from 0x03e363 to 0x00589b (0x38ac7 bytes) 000537 1787136531.452334 read 0x004a50 thru 0x0050c7 (0x678 bytes) 000538 1787136531.473699 write 0x026363 thru 0x032608 (0xc2a6 bytes) 000539 1787136531.476802 trunc from 0x032609 to 0x022579 (0x1008f bytes) 000540 1787136531.508204 mapwrite 0x00cebd thru 0x01a819 (0xd95d bytes) 000541 1787136531.562298 fallocate 0x039de1 thru 0x03a40b (0x62b bytes) 000542 1787136531.567944 write 0x02eb48 thru 0x033b2f (0x4fe8 bytes) 000543 1787136531.570106 trunc from 0x033b30 to 0x01c561 (0x175ce bytes) 000544 1787136531.595779 read 0x004877 thru 0x005c07 (0x1391 bytes) 000545 1787136531.597910 trunc from 0x01c562 to 0x01196e (0xabf3 bytes) 000546 1787136531.627428 trunc from 0x00262f to 0x00d33d (0xad0f bytes) 000547 1787136531.648343 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1196f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1196f): Mirror 3: 0x382759c0 (stale) Mirror 4: 0x382759c0 (stale) Mirror 5: 0x382759c0 (stale) Mirror 7: 0x382759c0 (stale) Mirror 9: 0x382759c0 (stale) Mirror 10: 0x382759c0 (stale) CRC-32 checksum value for chunk [0, 0x1196f): Mirror 1: 0xbdfc1385 Mirror 11: 0xbdfc1385 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1196f: skipped 000548 1787136532.758441 mapwrite 0x029773 thru 0x03519a (0xba28 bytes) 000549 1787136532.886253 read 0x01211b thru 0x01ba95 (0x997b bytes) 000551 1787136532.898307 read 0x02ef52 thru 0x03519a (0x6249 bytes) 000552 1787136532.899276 punch from 0x025be2 to 0x02bc79 (0x6098 bytes) 000553 1787136532.906002 read 0x02b31a thru 0x03519a (0x9e81 bytes) 000554 1787136532.924237 punch from 0x0140ca to 0x018b58 (0x4a8f bytes) 000555 1787136532.934676 mapwrite 0x005408 thru 0x00c95f (0x7558 bytes) 000556 1787136532.962664 fallocate 0x01030d thru 0x011de4 (0x1ad8 bytes) 000557 1787136532.977315 MIRROR_RESYNC 8 mirrors 000558 1787136535.419115 fallocate 0x0346c6 thru 0x03675e (0x2099 bytes) 000559 1787136535.481590 write 0x003f0a thru 0x00b3b9 (0x74b0 bytes) 000560 1787136535.501239 mapread 0x026a1e thru 0x026ae4 (0xc7 bytes) 000561 1787136535.510787 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3675f): Mirror 3: 0xeac77556 (stale) Mirror 4: 0xeac77556 (stale) Mirror 5: 0xeac77556 (stale) Mirror 7: 0xeac77556 (stale) Mirror 9: 0xeac77556 (stale) Mirror 10: 0xeac77556 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3675f): Mirror 3: 0xeac77556 (stale) Mirror 4: 0xeac77556 (stale) Mirror 5: 0xeac77556 (stale) Mirror 7: 0xeac77556 (stale) Mirror 9: 0xeac77556 (stale) Mirror 10: 0xeac77556 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3675f: skipped 000562 1787136536.351108 mapwrite 0x02f017 thru 0x037802 (0x87ec bytes) 000563 1787136536.454022 read 0x01644c thru 0x01eef5 (0x8aaa bytes) 000564 1787136536.475314 MIRROR_RESYNC 7 mirrors 000565 1787136538.287971 trunc from 0x012c94 to 0x01f445 (0xc7b2 bytes) 000566 1787136538.329632 MIRROR_RESYNC 7 mirrors 000567 1787136540.423500 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) 000568 1787136541.070329 trunc from 0x015a49 to 0x015ef7 (0x4af bytes) 000569 1787136541.130219 fallocate 0x03398f thru 0x03ffff (0xc671 bytes) 000570 1787136541.137284 punch from 0x02429d to 0x026fec (0x2d50 bytes) 000572 1787136541.142724 write 0x01dacf thru 0x024cb7 (0x71e9 bytes) 000573 1787136541.160037 trunc from 0x040000 to 0x0162d0 (0x29d2f bytes) 000574 1787136541.187404 punch from 0x002ec3 to 0x012d4f (0xfe8d bytes) 000575 1787136541.195215 mapread 0x00632a thru 0x0137bf (0xd496 bytes) 000576 1787136541.237696 mapwrite 0x00eddb thru 0x01363e (0x4864 bytes) 000577 1787136541.264613 mapread 0x004474 thru 0x00a1a4 (0x5d31 bytes) 000578 1787136541.272143 MIRROR_RESYNC 6 mirrors 000579 1787136542.782440 read 0x00157e thru 0x0091d4 (0x7c57 bytes) 000581 1787136542.799098 mapwrite 0x01054f thru 0x0176d3 (0x7185 bytes) 000582 1787136542.964497 write 0x026f21 thru 0x02ec12 (0x7cf2 bytes) 000583 1787136542.969750 trunc from 0x0248b3 to 0x025f4b (0x1699 bytes) 000584 1787136542.979365 read 0x01cef0 thru 0x01e8d3 (0x19e4 bytes) 000585 1787136543.015318 mapwrite 0x0372c7 thru 0x03f7c8 (0x8502 bytes) 000586 1787136543.149937 MIRROR_RESYNC 6 mirrors 000587 1787136545.503527 fallocate 0x0062cf thru 0x0139b9 (0xd6eb bytes) 000588 1787136545.540807 mapread 0x012e59 thru 0x012f90 (0x138 bytes) 000589 1787136545.548413 fallocate 0x034893 thru 0x03a17d (0x58eb bytes) 000590 1787136545.553683 trunc from 0x00aa17 to 0x01823e (0xd828 bytes) 000591 1787136545.560103 mapread 0x02f14a thru 0x03c875 (0xd72c bytes) 000592 1787136545.586155 trunc from 0x03f7c9 to 0x004be1 (0x3abe7 bytes) 000594 1787136545.612372 write 0x00fa6b thru 0x0186b2 (0x8c48 bytes) 000595 1787136545.614612 read 0x015754 thru 0x0186b2 (0x2f5f bytes) 000596 1787136545.616345 trunc from 0x007285 to 0x0161b5 (0xef31 bytes) 000597 1787136545.632338 trunc from 0x0186b3 to 0x02a596 (0x11ee4 bytes) 000599 1787136545.656772 fallocate 0x0355a8 thru 0x036e1e (0x1877 bytes) 000600 1787136545.663638 trunc from 0x02a597 to 0x03833a (0xdda4 bytes) 000601 1787136545.684086 fallocate 0x0398b4 thru 0x03ffff (0x674c bytes) 000603 1787136545.692381 trunc from 0x0182a7 to 0x024dc5 (0xcb1f bytes) 000604 1787136545.698678 mapwrite 0x031675 thru 0x03ffff (0xe98b bytes) 000605 1787136545.744349 fallocate 0x02833b thru 0x0340d6 (0xbd9c bytes) 000606 1787136545.753054 fallocate 0x028ebc thru 0x02ffe5 (0x712a bytes) 000608 1787136545.760384 fallocate 0x02d317 thru 0x034b3c (0x7826 bytes) 000609 1787136545.766061 fallocate 0x00229a thru 0x00fb24 (0xd88b bytes) 000610 1787136545.776372 MIRROR_RESYNC 6 mirrors 000611 1787136547.480145 read 0x024a3e thru 0x033278 (0xe83b bytes) 000612 1787136547.523038 mapread 0x018eea thru 0x01c894 (0x39ab bytes) 000613 1787136547.607957 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) 000614 1787136548.405030 fallocate 0x0097f7 thru 0x0186ac (0xeeb6 bytes) 000615 1787136548.476051 mapwrite 0x001666 thru 0x00d7f5 (0xc190 bytes) 000616 1787136548.571231 mapwrite 0x00becd thru 0x019d7d (0xdeb1 bytes) 000617 1787136548.651860 read 0x01e98d thru 0x0271a2 (0x8816 bytes) 000618 1787136548.665532 mapread 0x0097ed thru 0x00e611 (0x4e25 bytes) 000619 1787136548.673285 trunc from 0x040000 to 0x029c2f (0x163d0 bytes) 000620 1787136548.709198 mapwrite 0x034d5d thru 0x03ffff (0xb2a3 bytes) 000623 1787136548.828529 fallocate 0x0189a4 thru 0x01c562 (0x3bbf bytes) 000624 1787136548.837324 write 0x0258f7 thru 0x02b94e (0x6058 bytes) 000625 1787136548.848189 mapwrite 0x00cd4a thru 0x00f989 (0x2c40 bytes) 000626 1787136548.914993 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa08ba18e (stale) Mirror 7: 0xa08ba18e (stale) Mirror 9: 0xa08ba18e (stale) Mirror 10: 0xa08ba18e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc1f6f66 Mirror 11: 0xc1f6f66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000627 1787136550.249957 read 0x0198c1 thru 0x01f176 (0x58b6 bytes) 000628 1787136550.293582 write 0x029545 thru 0x029603 (0xbf bytes) 000629 1787136550.339525 mapread 0x01d698 thru 0x02702c (0x9995 bytes) 000631 1787136550.361409 mapwrite 0x026779 thru 0x026a12 (0x29a bytes) 000632 1787136550.376592 trunc from 0x0266c4 to 0x035518 (0xee55 bytes) 000633 1787136550.382969 read 0x006cb5 thru 0x014728 (0xda74 bytes) 000634 1787136550.396360 fallocate 0x01cf6c thru 0x025ceb (0x8d80 bytes) 000636 1787136550.402466 punch from 0x00d5e5 to 0x015aa4 (0x84c0 bytes) 000638 1787136550.410425 trunc from 0x00a751 to 0x01635d (0xbc0d bytes) 000639 1787136550.421150 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa08ba18e (stale) Mirror 9: 0xa08ba18e (stale) Mirror 10: 0xa08ba18e (stale) Mirror 11: 0xc1f6f66 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xa08ba18e (stale) Mirror 9: 0xa08ba18e (stale) Mirror 10: 0xa08ba18e (stale) Mirror 11: 0xc1f6f66 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000640 1787136550.960749 trunc from 0x00cbfc to 0x01910c (0xc511 bytes) 000641 1787136550.987186 write 0x014aa6 thru 0x01ced6 (0x8431 bytes) 000643 1787136551.031444 punch from 0x03be6b to 0x03d8ca (0x1a60 bytes) 000644 1787136551.045510 trunc from 0x040000 to 0x027a95 (0x1856a bytes) 000645 1787136551.070537 trunc from 0x027a96 to 0x0372f7 (0xf862 bytes) 000646 1787136551.102941 mapwrite 0x00cc53 thru 0x011b63 (0x4f11 bytes) 000647 1787136551.174423 trunc from 0x00db1a to 0x017ef1 (0xa3d8 bytes) 000649 1787136551.187496 mapwrite 0x02d6e6 thru 0x03b8eb (0xe206 bytes) 000651 1787136551.307000 punch from 0x026f32 to 0x02be75 (0x4f44 bytes) 000652 1787136551.331451 punch from 0x01bce5 to 0x01e07f (0x239b bytes) 000653 1787136551.340940 write 0x00c6cb thru 0x01b79e (0xf0d4 bytes) 000655 1787136551.359688 trunc from 0x032321 to 0x03733d (0x501d bytes) 000656 1787136551.369442 mapwrite 0x01ebf0 thru 0x025e3a (0x724b bytes) 000657 1787136551.444362 MIRROR_RESYNC 5 mirrors 000658 1787136553.465344 mapread 0x03aa31 thru 0x03b8eb (0xebb bytes) 000659 1787136553.477280 read 0x02e16e thru 0x0393ad (0xb240 bytes) 000661 1787136553.487811 mapwrite 0x00d7b7 thru 0x0177c5 (0xa00f bytes) 000662 1787136553.601536 mapwrite 0x0233eb thru 0x02be5f (0x8a75 bytes) 000663 1787136553.656440 MIRROR_RESYNC 5 mirrors 000665 1787136555.757795 write 0x00a28d thru 0x015bab (0xb91f bytes) 000666 1787136555.815460 fallocate 0x01fecc thru 0x0243cd (0x4502 bytes) 000667 1787136555.824342 read 0x024180 thru 0x032d7e (0xebff bytes) 000668 1787136555.835732 read 0x0029f3 thru 0x00cbc8 (0xa1d6 bytes) 000669 1787136555.848521 trunc from 0x00432f to 0x00f92c (0xb5fe bytes) 000670 1787136555.864419 trunc from 0x008d23 to 0x018537 (0xf815 bytes) 000671 1787136555.873186 fallocate 0x037a1a thru 0x0381a3 (0x78a bytes) 000672 1787136555.878801 write 0x039fe5 thru 0x03bf9d (0x1fb9 bytes) 000673 1787136555.880114 trunc from 0x03bf9e to 0x035041 (0x6f5c bytes) 000675 1787136555.907200 mapwrite 0x000d3b thru 0x0097c5 (0x8a8b bytes) 000676 1787136555.968724 mapread 0x0250e4 thru 0x02a6f4 (0x5611 bytes) 000677 1787136555.971013 mapwrite 0x00f3f5 thru 0x01170f (0x231b bytes) 000679 1787136556.008796 trunc from 0x026dcb to 0x035041 (0xe277 bytes) 000680 1787136556.021164 trunc from 0x035042 to 0x006331 (0x2ed10 bytes) 000681 1787136556.049259 fallocate 0x008d28 thru 0x012885 (0x9b5e bytes) 000682 1787136556.058338 trunc from 0x012886 to 0x0379e5 (0x25160 bytes) 000683 1787136556.090104 fallocate 0x014c08 thru 0x020da6 (0xc19f bytes) 000684 1787136556.096884 trunc from 0x0379e6 to 0x012940 (0x250a5 bytes) 000685 1787136556.125063 mapwrite 0x00b2c9 thru 0x010018 (0x4d50 bytes) 000686 1787136556.189683 fallocate 0x0149eb thru 0x0245aa (0xfbc0 bytes) 000687 1787136556.199358 fallocate 0x000aa9 thru 0x00d2bc (0xc814 bytes) 000688 1787136556.214156 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x245ab): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x245ab): Mirror 7: 0xe633f7e6 (stale) Mirror 9: 0xe633f7e6 (stale) Mirror 10: 0xe633f7e6 (stale) Mirror 11: 0xe633f7e6 (stale) CRC-32 checksum value for chunk [0, 0x245ab): Mirror 1: 0x9e126c99 Mirror 12: 0x9e126c99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x245ab: skipped 000689 1787136557.097746 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x245ab): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x245ab): Mirror 7: 0xe633f7e6 (stale) Mirror 9: 0xe633f7e6 (stale) Mirror 10: 0xe633f7e6 (stale) Mirror 11: 0xe633f7e6 (stale) CRC-32 checksum value for chunk [0, 0x245ab): Mirror 1: 0x9e126c99 Mirror 12: 0x9e126c99 Mirror 13: 0x9e126c99 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x245ab: skipped 000690 1787136558.147579 write 0x010904 thru 0x01e7d0 (0xdecd bytes) 000691 1787136558.239307 mapread 0x012376 thru 0x016b78 (0x4803 bytes) 000692 1787136558.240524 mapwrite 0x001840 thru 0x0071e0 (0x59a1 bytes) 000693 1787136558.323680 trunc from 0x00f080 to 0x01ef25 (0xfea6 bytes) 000694 1787136558.335883 trunc from 0x00f8b8 to 0x0148f1 (0x503a bytes) 000695 1787136558.352123 trunc from 0x0121dd to 0x01ceb0 (0xacd4 bytes) 000697 1787136558.374304 trunc from 0x00d28b to 0x00da56 (0x7cc bytes) 000698 1787136558.387051 mapwrite 0x02c27f thru 0x03528c (0x900e bytes) 000699 1787136558.512147 mapwrite 0x03aa53 thru 0x03ffff (0x55ad bytes) 000700 1787136558.588570 write 0x039085 thru 0x03ffff (0x6f7b bytes) 000701 1787136558.590344 write 0x00a2f2 thru 0x019713 (0xf422 bytes) 000702 1787136558.602531 write 0x033343 thru 0x03ed59 (0xba17 bytes) 000703 1787136558.605594 mapread 0x02685c thru 0x034b4e (0xe2f3 bytes) 000704 1787136558.607097 trunc from 0x011190 to 0x0114f8 (0x369 bytes) 000706 1787136558.631979 read 0x00eea2 thru 0x0148c9 (0x5a28 bytes) 000707 1787136558.638869 trunc from 0x040000 to 0x00b76e (0x34891 bytes) 000708 1787136558.663242 mapwrite 0x0036bd thru 0x00dea7 (0xa7eb bytes) 000709 1787136558.741184 mapwrite 0x010139 thru 0x018a51 (0x8919 bytes) 000710 1787136558.822886 trunc from 0x002fda to 0x01145f (0xe486 bytes) 000711 1787136558.832166 trunc from 0x018a52 to 0x004319 (0x14738 bytes) 000712 1787136558.856200 write 0x00d3f1 thru 0x017e1f (0xaa2f bytes) 000713 1787136558.868449 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17e20): Mirror 9: 0xdae6c5b5 (stale) Mirror 10: 0xdae6c5b5 (stale) Mirror 11: 0xdae6c5b5 (stale) Mirror 12: 0xffd9058a (stale) Mirror 13: 0xffd9058a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17e20): Mirror 9: 0xdae6c5b5 (stale) Mirror 10: 0xdae6c5b5 (stale) Mirror 11: 0xdae6c5b5 (stale) Mirror 12: 0xffd9058a (stale) Mirror 13: 0xffd9058a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17e20: skipped 000714 1787136559.616446 trunc from 0x017a9e to 0x017e1f (0x382 bytes) 000716 1787136559.622475 trunc from 0x00ef9f to 0x00f36e (0x3d0 bytes) 000717 1787136559.643685 write 0x0399e8 thru 0x03c55b (0x2b74 bytes) 000719 1787136559.653271 mapread 0x022a1d thru 0x0303a4 (0xd988 bytes) 000720 1787136559.684208 write 0x01b37a thru 0x024537 (0x91be bytes) 000721 1787136559.696250 MIRROR_RESYNC 6 mirrors 000722 1787136562.276925 mapread 0x007078 thru 0x00867b (0x1604 bytes) 000723 1787136562.306293 MIRROR_EXTEND to 7 mirrors 000724 1787136563.860716 read 0x02f6b4 thru 0x0364ae (0x6dfb bytes) 000726 1787136563.910091 punch from 0x0380b7 to 0x03c55b (0x44a5 bytes) 000727 1787136563.995785 trunc from 0x03c55c to 0x028af2 (0x13a69 bytes) 000728 1787136564.025541 fallocate 0x01bcd9 thru 0x02584d (0x9b75 bytes) 000729 1787136564.033422 mapwrite 0x02145b thru 0x02f65c (0xe202 bytes) 000730 1787136564.105678 mapread 0x025cfc thru 0x029ece (0x41d3 bytes) 000731 1787136564.109579 mapwrite 0x00b28e thru 0x014bea (0x995d bytes) 000732 1787136564.160458 mapwrite 0x027292 thru 0x02a54d (0x32bc bytes) 000733 1787136564.176370 trunc from 0x02f65d to 0x0080b5 (0x275a7 bytes) 000734 1787136564.195600 mapread 0x000da1 thru 0x001802 (0xa62 bytes) 000735 1787136564.206727 trunc from 0x001940 to 0x0080b5 (0x6776 bytes) 000736 1787136564.215587 trunc from 0x001471 to 0x0080b5 (0x6c45 bytes) 000737 1787136564.226724 write 0x037886 thru 0x03e38d (0x6b08 bytes) 000738 1787136564.237054 trunc from 0x031de3 to 0x03e38d (0xc5ab bytes) 000739 1787136564.267267 mapwrite 0x026592 thru 0x035727 (0xf196 bytes) 000740 1787136564.332792 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3e38e): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e38e): Mirror 9: 0x48ff8de0 (stale) Mirror 10: 0x48ff8de0 (stale) Mirror 11: 0x48ff8de0 (stale) Mirror 12: 0x48ff8de0 (stale) Mirror 13: 0x48ff8de0 (stale) Mirror 14: 0x48ff8de0 (stale) CRC-32 checksum value for chunk [0, 0x3e38e): Mirror 1: 0x5dc96e96 Mirror 15: 0x5dc96e96 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e38e: skipped 000741 1787136565.853023 fallocate 0x034b69 thru 0x03c7bd (0x7c55 bytes) 000742 1787136565.933791 write 0x00ec95 thru 0x018bd0 (0x9f3c bytes) 000743 1787136565.958963 mapread 0x0360e4 thru 0x036e60 (0xd7d bytes) 000744 1787136565.967072 trunc from 0x03e38e to 0x01e62b (0x1fd62 bytes) 000745 1787136566.038583 trunc from 0x01e62c to 0x0145bc (0xa06f bytes) 000746 1787136566.065777 punch from 0x00e5fd to 0x00fa68 (0x146c bytes) 000747 1787136566.084968 trunc from 0x009418 to 0x00f921 (0x650a bytes) 000748 1787136566.091128 mapwrite 0x01e141 thru 0x02d61f (0xf4df bytes) 000749 1787136566.156387 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d620): Mirror 10: 0xfe71f927 (stale) Mirror 11: 0xfe71f927 (stale) Mirror 12: 0xfe71f927 (stale) Mirror 13: 0xfe71f927 (stale) Mirror 14: 0xfe71f927 (stale) Mirror 15: 0x2b0a517e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d620): Mirror 10: 0xfe71f927 (stale) Mirror 11: 0xfe71f927 (stale) Mirror 12: 0xfe71f927 (stale) Mirror 13: 0xfe71f927 (stale) Mirror 14: 0xfe71f927 (stale) Mirror 15: 0x2b0a517e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d620: skipped 000750 1787136566.729443 trunc from 0x02d620 to 0x03d916 (0x102f7 bytes) 000751 1787136566.760909 fallocate 0x00ceb5 thru 0x011f9a (0x50e6 bytes) 000752 1787136566.766356 punch from 0x036700 to 0x03d2aa (0x6bab bytes) 000753 1787136566.771273 fallocate 0x03a420 thru 0x03ffff (0x5be0 bytes) 000754 1787136566.776486 punch from 0x0278a7 to 0x02f02a (0x7784 bytes) 000755 1787136566.781818 trunc from 0x03aeaa to 0x03ffff (0x5156 bytes) 000756 1787136566.793011 MIRROR_RESYNC 7 mirrors 000757 1787136569.639552 trunc from 0x0304ef to 0x03608d (0x5b9f bytes) 000758 1787136569.692738 read 0x0113b7 thru 0x01c63a (0xb284 bytes) 000759 1787136569.718088 trunc from 0x0330da to 0x03ffff (0xcf26 bytes) 000760 1787136569.731048 fallocate 0x012a5d thru 0x017386 (0x492a bytes) 000761 1787136569.745445 mapread 0x007bec thru 0x017656 (0xfa6b bytes) 000762 1787136569.785642 write 0x0209a0 thru 0x02b655 (0xacb6 bytes) 000763 1787136569.823023 write 0x01763b thru 0x02058c (0x8f52 bytes) 000764 1787136569.830679 punch from 0x03a7e6 to 0x03c4e0 (0x1cfb bytes) 000765 1787136569.839300 punch from 0x038d57 to 0x03e290 (0x553a bytes) 000766 1787136569.845233 mapread 0x02050c thru 0x02630c (0x5e01 bytes) 000767 1787136569.846521 mapwrite 0x0241f0 thru 0x025075 (0xe86 bytes) 000768 1787136569.909118 MIRROR_RESYNC 7 mirrors 000770 1787136572.228807 write 0x03979e thru 0x03ffff (0x6862 bytes) 000771 1787136572.261946 mapread 0x03e745 thru 0x03ebbe (0x47a bytes) 000772 1787136572.266073 read 0x026d8e thru 0x035b13 (0xed86 bytes) 000773 1787136572.275171 trunc from 0x040000 to 0x01c59e (0x23a61 bytes) 000774 1787136572.304994 mapread 0x0083a1 thru 0x011347 (0x8fa7 bytes) 000775 1787136572.329379 trunc from 0x01c59f to 0x00391e (0x18c80 bytes) 000776 1787136572.358084 read 0x000489 thru 0x00391e (0x3496 bytes) 000777 1787136572.387664 trunc from 0x00065d to 0x00391e (0x32c2 bytes) 000778 1787136572.400251 mapread 0x0016ef thru 0x001bea (0x4fc bytes) 000779 1787136572.421312 fallocate 0x006ef0 thru 0x00dca3 (0x6db4 bytes) 000780 1787136572.447561 punch from 0x00ae0a to 0x00dca3 (0x2e9a bytes) 000781 1787136572.468002 read 0x006965 thru 0x00ae0b (0x44a7 bytes) 000783 1787136572.499586 punch from 0x007784 to 0x00dca3 (0x6520 bytes) 000784 1787136572.510502 fallocate 0x029ca7 thru 0x038aff (0xee59 bytes) 000785 1787136572.517499 fallocate 0x01812d thru 0x01a825 (0x26f9 bytes) 000786 1787136572.525873 fallocate 0x01054f thru 0x01f12f (0xebe1 bytes) 000787 1787136572.534712 write 0x03f4c3 thru 0x03ffff (0xb3d bytes) 000788 1787136572.537060 write 0x03ff74 thru 0x03ffff (0x8c bytes) 000790 1787136572.538536 trunc from 0x040000 to 0x0110b4 (0x2ef4b bytes) 000791 1787136572.555657 read 0x00bcb8 thru 0x00f3c1 (0x370a bytes) 000792 1787136572.569472 punch from 0x00c125 to 0x0110b4 (0x4f90 bytes) 000793 1787136572.573955 read 0x007f77 thru 0x00d1f9 (0x5283 bytes) 000794 1787136572.585987 fallocate 0x02ef9e thru 0x03ad0c (0xbd6f bytes) 000795 1787136572.591945 read 0x00103f thru 0x0068b2 (0x5874 bytes) 000796 1787136572.592974 mapread 0x008376 thru 0x008d95 (0xa20 bytes) 000797 1787136572.595108 trunc from 0x008ab4 to 0x0110b4 (0x8601 bytes) 000798 1787136572.599887 fallocate 0x01560f thru 0x0193fd (0x3def bytes) 000799 1787136572.609373 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x110b5): Mirror 10: 0xed24d461 (stale) Mirror 11: 0xed24d461 (stale) Mirror 12: 0xed24d461 (stale) Mirror 14: 0xed24d461 (stale) Mirror 15: 0xed24d461 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x110b5): Mirror 10: 0xed24d461 (stale) Mirror 11: 0xed24d461 (stale) Mirror 12: 0xed24d461 (stale) Mirror 14: 0xed24d461 (stale) Mirror 15: 0xed24d461 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x110b5: skipped 000800 1787136573.064854 trunc from 0x0017e4 to 0x00f032 (0xd84f bytes) 000801 1787136573.072976 trunc from 0x00338f to 0x00afd5 (0x7c47 bytes) 000802 1787136573.080144 punch from 0x00604b to 0x00ff9a (0x9f50 bytes) 000803 1787136573.090654 read 0x00bb62 thru 0x0110b4 (0x5553 bytes) 000804 1787136573.113892 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x110b5): Mirror 10: 0xed24d461 (stale) Mirror 11: 0xed24d461 (stale) Mirror 14: 0xed24d461 (stale) Mirror 15: 0xed24d461 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x110b5): Mirror 10: 0xed24d461 (stale) Mirror 11: 0xed24d461 (stale) Mirror 14: 0xed24d461 (stale) Mirror 15: 0xed24d461 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x110b5: skipped 000805 1787136573.674928 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x110b5): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x110b5): Mirror 10: 0xed24d461 (stale) Mirror 11: 0xed24d461 (stale) Mirror 14: 0xed24d461 (stale) Mirror 15: 0xed24d461 (stale) CRC-32 checksum value for chunk [0, 0x110b5): Mirror 1: 0x35e0de24 Mirror 16: 0x35e0de24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x110b5: skipped 000806 1787136575.460450 MIRROR_RESYNC 6 mirrors 000807 1787136577.033128 fallocate 0x0206b4 thru 0x029bef (0x953c bytes) 000808 1787136577.084909 trunc from 0x00a902 to 0x0110b4 (0x67b3 bytes) 000809 1787136577.106919 trunc from 0x0110b5 to 0x02bb00 (0x1aa4c bytes) 000810 1787136577.149152 write 0x022f8c thru 0x02d105 (0xa17a bytes) 000811 1787136577.165176 trunc from 0x029d32 to 0x02d105 (0x33d4 bytes) 000812 1787136577.216883 trunc from 0x013728 to 0x016335 (0x2c0e bytes) 000813 1787136577.223033 trunc from 0x0156d7 to 0x01660c (0xf36 bytes) 000814 1787136577.232412 trunc from 0x02d106 to 0x000c85 (0x2c480 bytes) 000815 1787136577.282081 read 0x000b2b thru 0x000c85 (0x15b bytes) 000816 1787136577.295370 fallocate 0x015f16 thru 0x01fc02 (0x9ced bytes) 000817 1787136577.314191 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fc03): Mirror 10: 0x4a18ee61 (stale) Mirror 14: 0x4a18ee61 (stale) Mirror 15: 0x4a18ee61 (stale) Mirror 16: 0x4a18ee61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fc03): Mirror 10: 0x4a18ee61 (stale) Mirror 14: 0x4a18ee61 (stale) Mirror 15: 0x4a18ee61 (stale) Mirror 16: 0x4a18ee61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fc03: skipped 000818 1787136578.218023 read 0x0024c5 thru 0x0086e3 (0x621f bytes) 000819 1787136578.237082 punch from 0x0114ef to 0x01fc02 (0xe714 bytes) 000821 1787136578.248926 fallocate 0x037f5b thru 0x03a74a (0x27f0 bytes) 000822 1787136578.257162 read 0x0243f1 thru 0x03257c (0xe18c bytes) 000823 1787136578.299471 mapread 0x022912 thru 0x02e8ac (0xbf9b bytes) 000824 1787136578.303072 mapread 0x02d9fe thru 0x030b3a (0x313d bytes) 000825 1787136578.305056 mapread 0x00ed08 thru 0x018465 (0x975e bytes) 000826 1787136578.306654 write 0x034731 thru 0x03d4f9 (0x8dc9 bytes) 000827 1787136578.314423 read 0x0353e1 thru 0x038ffb (0x3c1b bytes) 000829 1787136578.316946 fallocate 0x033998 thru 0x036cf2 (0x335b bytes) 000830 1787136578.324691 trunc from 0x03d4fa to 0x01ea06 (0x1eaf3 bytes) 000831 1787136578.353696 MIRROR_RESYNC 5 mirrors 000832 1787136579.686701 mapwrite 0x00e218 thru 0x011ce6 (0x3acf bytes) 000833 1787136579.776760 mapread 0x01b593 thru 0x01cfd3 (0x1a41 bytes) 000834 1787136579.788261 mapread 0x003105 thru 0x01153a (0xe436 bytes) 000835 1787136579.802203 write 0x01318f thru 0x013235 (0xa7 bytes) 000836 1787136579.807517 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1ea07): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ea07): Mirror 10: 0x9eec6837 (stale) Mirror 14: 0x9eec6837 (stale) Mirror 15: 0x9eec6837 (stale) Mirror 16: 0x9eec6837 (stale) CRC-32 checksum value for chunk [0, 0x1ea07): Mirror 1: 0x7417ed37 Mirror 17: 0x7417ed37 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ea07: skipped 000837 1787136581.092880 mapwrite 0x02d68f thru 0x03d141 (0xfab3 bytes) 000838 1787136581.390402 mapwrite 0x001ce3 thru 0x00e4d7 (0xc7f5 bytes) 000839 1787136581.450689 MIRROR_RESYNC 6 mirrors 000841 1787136583.595700 mapread 0x00e797 thru 0x01c1df (0xda49 bytes) 000843 1787136583.637750 trunc from 0x03d142 to 0x015e44 (0x272fd bytes) 000844 1787136583.699079 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15e45): Mirror 10: 0x6bf55aca (stale) Mirror 14: 0x6bf55aca (stale) Mirror 15: 0x6bf55aca (stale) Mirror 17: 0x6bf55aca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15e45): Mirror 10: 0x6bf55aca (stale) Mirror 14: 0x6bf55aca (stale) Mirror 15: 0x6bf55aca (stale) Mirror 17: 0x6bf55aca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15e45: skipped 000845 1787136584.232847 punch from 0x014b70 to 0x015e44 (0x12d5 bytes) 000846 1787136584.239763 fallocate 0x036c2b thru 0x03ffff (0x93d5 bytes) 000847 1787136584.245506 trunc from 0x030cf5 to 0x03f13e (0xe44a bytes) 000850 1787136584.252690 punch from 0x031b8c to 0x03ad06 (0x917b bytes) 000852 1787136584.259197 write 0x031342 thru 0x03ead6 (0xd795 bytes) 000854 1787136584.286321 mapread 0x0102b8 thru 0x01a66c (0xa3b5 bytes) 000856 1787136584.333082 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x5f33d696 (stale) Mirror 14: 0x5f33d696 (stale) Mirror 15: 0x5f33d696 (stale) Mirror 17: 0x5f33d696 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x39fdc2a1 Mirror 18: 0x39fdc2a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000857 1787136585.243560 fallocate 0x033687 thru 0x0358e0 (0x225a bytes) 000858 1787136585.331950 trunc from 0x02d296 to 0x039ea3 (0xcc0e bytes) 000859 1787136585.339913 mapread 0x035254 thru 0x0363ea (0x1197 bytes) 000860 1787136585.367816 trunc from 0x040000 to 0x006a8b (0x39574 bytes) 000861 1787136585.447679 trunc from 0x006a8c to 0x026079 (0x1f5ee bytes) 000863 1787136585.494214 MIRROR_RESYNC 6 mirrors 000864 1787136586.661474 trunc from 0x021718 to 0x026079 (0x4962 bytes) 000865 1787136586.698406 mapwrite 0x01ab4a thru 0x025728 (0xabdf bytes) 000866 1787136586.792809 read 0x00c980 thru 0x012503 (0x5b84 bytes) 000867 1787136586.823178 fallocate 0x0106ca thru 0x01e6af (0xdfe6 bytes) 000868 1787136586.831019 read 0x00ce78 thru 0x00dee0 (0x1069 bytes) 000869 1787136586.841651 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2607a): Mirror 10: 0xa66994bb (stale) Mirror 14: 0xa66994bb (stale) Mirror 15: 0xa66994bb (stale) Mirror 17: 0xa66994bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2607a): Mirror 10: 0xa66994bb (stale) Mirror 14: 0xa66994bb (stale) Mirror 15: 0xa66994bb (stale) Mirror 17: 0xa66994bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2607a: skipped 000870 1787136587.304947 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2607a): Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2607a): Mirror 10: 0xa66994bb (stale) Mirror 14: 0xa66994bb (stale) Mirror 15: 0xa66994bb (stale) Mirror 17: 0xa66994bb (stale) CRC-32 checksum value for chunk [0, 0x2607a): Mirror 1: 0x8d51ac13 Mirror 18: 0x8d51ac13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2607a: skipped 000871 1787136588.447337 mapwrite 0x010611 thru 0x01b9e5 (0xb3d5 bytes) 000872 1787136588.707451 mapwrite 0x038423 thru 0x03a0f0 (0x1cce bytes) 000873 1787136588.808917 trunc from 0x03a0f1 to 0x006c22 (0x334ce bytes) 000874 1787136588.833356 write 0x02e0da thru 0x02f7fc (0x1723 bytes) 000875 1787136588.834566 trunc from 0x02f7fd to 0x01f23e (0x105be bytes) 000876 1787136588.856975 mapwrite 0x039c10 thru 0x03ffff (0x63f0 bytes) 000877 1787136588.937580 trunc from 0x00eeaa to 0x01d63f (0xe796 bytes) 000878 1787136588.945101 punch from 0x02a352 to 0x02b18f (0xe3e bytes) 000879 1787136588.954986 trunc from 0x040000 to 0x01c4c1 (0x23b3e bytes) 000880 1787136588.999699 write 0x017168 thru 0x0267d4 (0xf66d bytes) 000881 1787136589.015881 punch from 0x01d3c2 to 0x01e3a2 (0xfe1 bytes) 000882 1787136589.047105 trunc from 0x01fa1f to 0x0267d4 (0x6db6 bytes) 000883 1787136589.056897 trunc from 0x0061dc to 0x006fb4 (0xdd9 bytes) 000884 1787136589.068575 read 0x003c7a thru 0x013145 (0xf4cc bytes) 000885 1787136589.099599 punch from 0x020ac2 to 0x0267d4 (0x5d13 bytes) 000886 1787136589.118167 fallocate 0x01ab85 thru 0x0277ed (0xcc69 bytes) 000887 1787136589.126456 trunc from 0x0267d5 to 0x00eb06 (0x17cce bytes) 000888 1787136589.152616 read 0x00a37f thru 0x00e742 (0x43c4 bytes) 000889 1787136589.165653 MIRROR_RESYNC 6 mirrors 000890 1787136590.600882 fallocate 0x00fd32 thru 0x01e74e (0xea1d bytes) 000891 1787136590.652412 trunc from 0x00ea5f to 0x00eb06 (0xa8 bytes) 000892 1787136590.657887 write 0x028776 thru 0x033dab (0xb636 bytes) 000893 1787136590.659633 punch from 0x029867 to 0x033dab (0xa545 bytes) 000894 1787136590.681578 trunc from 0x02505a to 0x02efd6 (0x9f7d bytes) 000895 1787136590.687742 trunc from 0x033dac to 0x029f3a (0x9e71 bytes) 000896 1787136590.712657 write 0x009d2e thru 0x018be7 (0xeeba bytes) 000897 1787136590.729757 trunc from 0x029f3b to 0x034d24 (0xadea bytes) 000898 1787136590.761928 trunc from 0x00afa6 to 0x013ab4 (0x8b0f bytes) 000899 1787136590.784572 trunc from 0x034d25 to 0x038bc1 (0x3e9d bytes) 000900 1787136590.806463 read 0x024b99 thru 0x02c24f (0x76b7 bytes) 000902 1787136590.823817 trunc from 0x038bc2 to 0x0330a8 (0x5b19 bytes) 000903 1787136590.842565 mapread 0x0049f6 thru 0x007a19 (0x3024 bytes) 000904 1787136590.866684 write 0x03aea8 thru 0x03defd (0x3056 bytes) 000905 1787136590.868140 read 0x02d209 thru 0x0318ce (0x46c6 bytes) 000906 1787136590.882372 mapwrite 0x002213 thru 0x0033cd (0x11bb bytes) 000907 1787136590.905519 trunc from 0x001009 to 0x0029f1 (0x19e9 bytes) 000908 1787136590.921240 MIRROR_RESYNC 6 mirrors 000909 1787136593.358668 read 0x013b28 thru 0x02122b (0xd704 bytes) 000910 1787136593.371504 write 0x0338f8 thru 0x03a229 (0x6932 bytes) 000911 1787136593.417016 fallocate 0x01c962 thru 0x0281b1 (0xb850 bytes) 000912 1787136593.426011 fallocate 0x0126a3 thru 0x018846 (0x61a4 bytes) 000913 1787136593.432423 write 0x037800 thru 0x03ffff (0x8800 bytes) 000914 1787136593.434781 trunc from 0x040000 to 0x01769e (0x28961 bytes) 000915 1787136593.459250 mapwrite 0x02842b thru 0x033395 (0xaf6b bytes) 000916 1787136593.592901 mapread 0x02d1f0 thru 0x033395 (0x61a6 bytes) 000917 1787136593.595616 trunc from 0x02dc30 to 0x0324ff (0x48d0 bytes) 000918 1787136593.601518 mapwrite 0x00f295 thru 0x01444e (0x51ba bytes) 000919 1787136593.627733 mapread 0x0270c6 thru 0x033395 (0xc2d0 bytes) 000920 1787136593.637662 trunc from 0x017487 to 0x0179dc (0x556 bytes) 000921 1787136593.644010 write 0x01c6af thru 0x023274 (0x6bc6 bytes) 000922 1787136593.650539 read 0x0232a6 thru 0x02bf40 (0x8c9b bytes) 000923 1787136593.652141 trunc from 0x033396 to 0x028d41 (0xa654 bytes) 000924 1787136593.701736 MIRROR_RESYNC 6 mirrors 000925 1787136596.212511 MIRROR_EXTEND to 7 mirrors 000926 1787136597.425910 punch from 0x00d0ae to 0x012b97 (0x5aea bytes) 000927 1787136597.488111 mapwrite 0x0043b3 thru 0x00532b (0xf79 bytes) 000928 1787136597.540704 trunc from 0x028d42 to 0x028220 (0xb21 bytes) 000929 1787136597.593629 write 0x013293 thru 0x020850 (0xd5be bytes) 000930 1787136597.638069 punch from 0x01d87d to 0x01e92b (0x10af bytes) 000931 1787136597.676159 fallocate 0x0052b1 thru 0x00a0d0 (0x4e20 bytes) 000932 1787136597.683088 punch from 0x02737c to 0x028220 (0xea5 bytes) 000933 1787136597.695494 trunc from 0x008229 to 0x014250 (0xc028 bytes) 000934 1787136597.704795 mapwrite 0x0125ac thru 0x01e000 (0xba55 bytes) 000935 1787136597.787695 punch from 0x01fac2 to 0x028220 (0x875f bytes) 000936 1787136597.798507 punch from 0x006aeb to 0x0112b2 (0xa7c8 bytes) 000937 1787136597.811551 read 0x015e24 thru 0x016221 (0x3fe bytes) 000938 1787136597.813612 trunc from 0x0231ea to 0x027918 (0x472f bytes) 000939 1787136597.849945 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x16a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28221): Mirror 14: 0x2d688293 (stale) Mirror 15: 0x2d688293 (stale) Mirror 17: 0x2d688293 (stale) Mirror 18: 0x2d688293 (stale) Mirror 19: 0x2d688293 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28221): Mirror 14: 0x2d688293 (stale) Mirror 15: 0x2d688293 (stale) Mirror 17: 0x2d688293 (stale) Mirror 18: 0x2d688293 (stale) Mirror 19: 0x2d688293 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28221: skipped 000940 1787136598.520320 trunc from 0x028221 to 0x016a03 (0x1181d bytes) 000941 1787136598.549924 mapwrite 0x005600 thru 0x00f187 (0x9b88 bytes) 000942 1787136598.598655 write 0x01d6b5 thru 0x029cd2 (0xc61e bytes) 000943 1787136598.600458 trunc from 0x014d8d to 0x01d7fe (0x8a72 bytes) 000944 1787136598.618681 mapread 0x021355 thru 0x028010 (0x6cbc bytes) 000945 1787136598.625610 MIRROR_RESYNC 6 mirrors 000946 1787136600.321697 mapread 0x0286c4 thru 0x029cd2 (0x160f bytes) 000947 1787136600.345680 punch from 0x009dc2 to 0x00ed13 (0x4f52 bytes) 000948 1787136600.405353 mapwrite 0x02722f thru 0x032230 (0xb002 bytes) 000949 1787136600.489009 trunc from 0x02ae74 to 0x032230 (0x73bd bytes) 000950 1787136600.499561 mapwrite 0x03f947 thru 0x03ffff (0x6b9 bytes) 000951 1787136600.549416 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x69f6a55e (stale) Mirror 15: 0x69f6a55e (stale) Mirror 17: 0x69f6a55e (stale) Mirror 18: 0x69f6a55e (stale) Mirror 19: 0x69f6a55e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xee040458 Mirror 20: 0xee040458 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000952 1787136601.965733 write 0x038555 thru 0x03e881 (0x632d bytes) 000953 1787136602.083679 fallocate 0x015233 thru 0x017dbb (0x2b89 bytes) 000954 1787136602.114600 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x69f6a55e (stale) Mirror 15: 0x69f6a55e (stale) Mirror 17: 0x69f6a55e (stale) Mirror 18: 0x69f6a55e (stale) Mirror 19: 0x69f6a55e (stale) Mirror 20: 0xee040458 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6f56068a Mirror 21: 0x6f56068a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000955 1787136603.831088 write 0x000da1 thru 0x002c0a (0x1e6a bytes) 000956 1787136603.930790 trunc from 0x040000 to 0x005649 (0x3a9b6 bytes) 000957 1787136603.965624 mapread 0x003928 thru 0x005649 (0x1d22 bytes) 000958 1787136603.991351 fallocate 0x01ef33 thru 0x02ea8d (0xfb5b bytes) 000959 1787136604.002363 punch from 0x02657b to 0x028165 (0x1beb bytes) 000960 1787136604.007271 fallocate 0x034953 thru 0x03ffff (0xb6ad bytes) 000962 1787136604.013985 punch from 0x00a19c to 0x016a4f (0xc8b4 bytes) 000964 1787136604.021630 fallocate 0x022905 thru 0x029c49 (0x7345 bytes) 000965 1787136604.032291 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2ea8e): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ea8e): Mirror 14: 0x37e7f29 (stale) Mirror 15: 0x37e7f29 (stale) Mirror 17: 0x37e7f29 (stale) Mirror 18: 0x37e7f29 (stale) Mirror 19: 0x37e7f29 (stale) Mirror 20: 0x9e00772b (stale) Mirror 21: 0x9e00772b (stale) CRC-32 checksum value for chunk [0, 0x2ea8e): Mirror 1: 0x4ebdadf Mirror 22: 0x4ebdadf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ea8e: skipped 000966 1787136605.626671 mapwrite 0x0263fc thru 0x031d68 (0xb96d bytes) 000967 1787136605.789741 mapwrite 0x024fa5 thru 0x028d57 (0x3db3 bytes) 000968 1787136605.810696 mapwrite 0x030972 thru 0x03bfa0 (0xb62f bytes) 000969 1787136605.885413 read 0x033e0b thru 0x03a22b (0x6421 bytes) 000971 1787136605.887633 trunc from 0x0234f3 to 0x030883 (0xd391 bytes) 000973 1787136605.895460 read 0x0255ce thru 0x02bf17 (0x694a bytes) 000974 1787136605.934129 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3bfa1): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfa1): Mirror 14: 0xfbbd8e6f (stale) Mirror 15: 0xfbbd8e6f (stale) Mirror 17: 0xfbbd8e6f (stale) Mirror 18: 0xfbbd8e6f (stale) Mirror 19: 0xfbbd8e6f (stale) Mirror 20: 0x53586113 (stale) Mirror 21: 0xfdad2fcf (stale) Mirror 22: 0x806f1244 (stale) CRC-32 checksum value for chunk [0, 0x3bfa1): Mirror 1: 0x20e9a971 Mirror 23: 0x20e9a971 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfa1: skipped 000975 1787136607.545521 mapwrite 0x009888 thru 0x012161 (0x88da bytes) 000976 1787136607.741209 punch from 0x0197de to 0x024ce1 (0xb504 bytes) 000977 1787136607.749431 read 0x0103f2 thru 0x01fc94 (0xf8a3 bytes) 000978 1787136607.772150 trunc from 0x0096aa to 0x015817 (0xc16e bytes) 000979 1787136607.782358 trunc from 0x01b36a to 0x028fcf (0xdc66 bytes) 000980 1787136607.791735 read 0x01cdc6 thru 0x0293a0 (0xc5db bytes) 000982 1787136607.816492 read 0x0239ff thru 0x030fdc (0xd5de bytes) 000983 1787136607.823092 mapread 0x0057e4 thru 0x011e83 (0xc6a0 bytes) 000985 1787136607.825328 fallocate 0x0016e1 thru 0x01159b (0xfebb bytes) 000989 1787136607.846456 trunc from 0x03bfa1 to 0x038c37 (0x3369 bytes) 000990 1787136607.913153 MIRROR_RESYNC 10 mirrors 000991 1787136611.060043 read 0x024849 thru 0x02e0c1 (0x9879 bytes) 000992 1787136611.071558 punch from 0x034ee2 to 0x038c37 (0x3d56 bytes) 000993 1787136611.140677 punch from 0x02ee26 to 0x033bad (0x4d88 bytes) 000994 1787136611.149363 write 0x024362 thru 0x02e1c9 (0x9e68 bytes) 000996 1787136611.185875 read 0x03715a thru 0x038c37 (0x1ade bytes) 000997 1787136611.215574 write 0x0298df thru 0x034ec9 (0xb5eb bytes) 000998 1787136611.230236 fallocate 0x0094c4 thru 0x00fe3d (0x697a bytes) 000999 1787136611.247463 trunc from 0x038c38 to 0x037b8d (0x10aa bytes) All operations completed A-OK! PASS 70a (142s) == sanity-flr test 71: check mirror extend parent fid ==== 06:50:18 (1787136618) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x55:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (15s) == sanity-flr test 72: check append on FLR file ========== 06:50:34 (1787136634) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164095 s, 12.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975 s, 10.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 06:50:43 (1787136643) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 623296 .. Extending file size to 5059104 .. failed Extending file size to 8153728 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5969280 .. Extending file size to 8652832 .. Extending file size to 8885984 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9063168 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61361 61363 61364 61365 61367 done 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) 192.168.204.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:52:10 (1787136730) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 835200 .. done Extending file size to 4226752 .. Extending file size to 1446720 .. Extending file size to 2291520 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7801760 .. Extending file size to 8630304 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8966656 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8564864 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62743 62745 62746 62748 62749 done 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) 192.168.204.120@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (95s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:53:46 (1787136826) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2224 1269080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25284 3581736 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17552 3589468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42836 7171204 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2224 1269080 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25284 3581736 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17552 3589468 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42836 7171204 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.201583 s, 10.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.255644 s, 8.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd85eb9be (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:54:09 (1787136849) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20597 s, 10.2 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 06:54:18 (1787136858) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:54:28 (1787136868) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14458 s, 14.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:54:38 (1787136878) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149234 s, 14.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:54:48 (1787136888) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0839033 s, 25.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:55:23 (1787136923) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851764 s, 12.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x211:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x210:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x211:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x210:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x210:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:56:18 (1787136978) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0941485 s, 22.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:57:11 (1787137031) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg420-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg420-server: Pool lustre.test_204f created Adding targets to pool oleg420-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg420-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134777 s, 15.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg420-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg420-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg420-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:58:34 (1787137114) PASS 205a (11s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:58:45 (1787137125) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:58:55 (1787137135) oleg420-server: Pool lustre.test_206 created Adding targets to pool oleg420-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg420-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg420-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg420-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg420-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:59:24 (1787137164) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943727 s, 11.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390593 s, 26.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf68387fd Mirror 3: 0xf68387fd PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:59:36 (1787137176) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:59:40 (1787137180) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:59:44 (1787137184) Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0624874 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1010000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xfc0000 Read this (4 bytes): BBBB 192.168.204.120@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg420-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:00:22 (1787137222) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0198727 s, 412 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:00:30 (1787137230) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444739 s, 23.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x87 lmm_fid: [0x200000405:0x87:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x87 lmm_fid: [0x200000405:0x87:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x121:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611271 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-19 07:00:32.000000000 -0400 Modify: 2026-08-19 07:00:32.000000000 -0400 Change: 2026-08-19 07:00:34.000000000 -0400 Birth: 2026-08-19 07:00:32.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:00:40 (1787137240) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460422 s, 22.8 MB/s oleg420-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg420-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg420-server: O/280000401/d31/13f: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:01:05 (1787137265) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.258817 s, 158 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0227899 s, 180 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 19 07:01 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 19 07:01 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:01:14 (1787137274) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:01:22 (1787137282) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 2033 sec =========== 07:01:32 (1787137292) === sanity-flr: start cleanup 07:01:34 (1787137294) === === sanity-flr: finish cleanup 07:01:43 (1787137303) ===