-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 21:12:59 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg440-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 21:13:20 (1787793200) === oleg440-client.virtnet: executing check_config_client /mnt/lustre oleg440-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg440-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a93d106c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a93d106c000.idle_timeout=debug disable quota as required oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 21:14:06 (1787793246) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8238] == sanity-flr test 0a: lfs mirror create with -N option == 21:14:09 (1787793249) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 21:14:29 (1787793269) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 21:14:33 (1787793273) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 21:14:36 (1787793276) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 21:15:03 (1787793303) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 21:15:07 (1787793307) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 21:15:11 (1787793311) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8a93d106c000.stats=clear osc.lustre-OST0001-osc-ffff8a93d106c000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 21:15:19 (1787793319) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 21:15:31 (1787793331) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 21:15:40 (1787793340) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 21:15:43 (1787793343) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 21:15:53 (1787793353) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 21:16:04 (1787793364) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 21:16:11 (1787793371) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 21:16:20 (1787793380) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 21:16:28 (1787793388) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 21:16:39 (1787793399) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 21:16:47 (1787793407) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.440235 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632519 s, 16.6 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 21:16:56 (1787793416) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.562933 s, 31.7 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 21:17:05 (1787793425) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 21:17:15 (1787793435) Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a93d106c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a93d106c000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg440-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a93d106c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a93d106c000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg440-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (137s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 21:19:32 (1787793572) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.309728 s, 13.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a93d106c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a93d106c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 21:20:20 (1787793620) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a93d106c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a93d106c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a93d106c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a93d106c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg440-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (79s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 21:21:40 (1787793700) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.84117 s, 18.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 21:21 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.946s user 0m0.001s sys 0m0.183s Stopping /mnt/lustre-ost1 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a93d106c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a93d106c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.357s user 0m0.000s sys 0m0.187s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg440-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg440-server oleg440-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a93d106c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a93d106c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.264s user 0m0.000s sys 0m0.200s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg440-server: oleg440-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg440-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg440-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a93d106c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a93d106c000.ost_server_uuid in FULL state after 0 sec PASS 33b (87s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 21:23:07 (1787793787) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 21:23:10 (1787793790) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 21:23:14 (1787793794) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 21:23:18 (1787793798) PASS 35 (11s) == sanity-flr test 36a: write to mirrored files ========== 21:23:29 (1787793809) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.641278 s, 19.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (17s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 21:23:46 (1787793826) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532391 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366614 s, 28.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 21:23:55 (1787793835) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 21:24:03 (1787793843) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00849975 s, 120 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00170429 s, 601 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00193532 s, 529 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00099268 s, 1.0 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 21:24:14 (1787793854) create test files with size 15749934 .. d9b7f5d87d9fbfbb32d5492ab6669c35 - 4788d4bd846f09b9b57099d002101296 - f763eec442f4787127b4e960056f882a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg440-client.virtnet /mnt/lustre (opts:) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 21:24:38 (1787793878) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 21:24:54 (1787793894) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 21:25:02 (1787793902) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824658 s, 12.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026299 s, 39.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 21:25:14 (1787793914) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159066 s, 13.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.377762 s, 11.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203598 s, 10.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 21:25:34 (1787793934) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 21:25:37 (1787793937) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 21:25:41 (1787793941) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416635 s, 25.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 43c: read prefer uncompressed mirror == 21:25:48 (1787793948) llite.lustre-ffff8a93c88d7800.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (6s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 21:25:54 (1787793954) llite.lustre-ffff8a93c88d7800.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308512 s, 34.0 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297485 s, 35.2 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 21:26:02 (1787793962) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.264597 s, 11.9 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 21:26:12 (1787793972) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xce:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 21:26:23 (1787793983) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.361555 s, 29.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg440-client.virtnet /mnt/lustre (opts:) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 21:26:36 (1787793996) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8a93c6be4000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.415354 s, 25.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg440-client.virtnet /mnt/lustre (opts:) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8a93c2ae1800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 21:26:54 (1787794014) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 21:27:33 (1787794053) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 21:28:09 (1787794089) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 21:28:13 (1787794093) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.338182 s, 9.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.132301 s, 23.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa7697a36 (nosync,stale) Mirror 2: 0xa7697a36 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7cfc8b4c415c591ad46f57dc63f2c2cb - mirror 1 checksum: 3355b037ebc52f192bd91c7c5c87ffa8 - mirror 2 checksum: 7cfc8b4c415c591ad46f57dc63f2c2cb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T21:28:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T21:28:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7cfc8b4c415c591ad46f57dc63f2c2cb - mirror 1 checksum: 3355b037ebc52f192bd91c7c5c87ffa8 - mirror 2 checksum: 3355b037ebc52f192bd91c7c5c87ffa8 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T21:28:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T21:28:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xdc:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 21:28:26 (1787794106) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647199 s, 16.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 21:28:35 (1787794115) Failing mds1 on oleg440-server Stopping /mnt/lustre-mds1 (opts:) on oleg440-server 21:28:47 (1787794127) shut down facet: mds1 facet_host: oleg440-server facet_failover_host: oleg440-server Failover mds1 to oleg440-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 21:29:13 (1787794153) targets are mounted 21:29:13 (1787794153) facet_failover done oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376881 s, 27.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 50A (57s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 21:29:33 (1787794173) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0553726 s, 370 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0138939 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 21:29:46 (1787794186) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.109018 s, 188 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0154116 s, 1.3 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 21:29:57 (1787794197) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0144345 s, 1.1 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 21:30:05 (1787794205) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0558948 s, 366 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 21:30:14 (1787794214) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00944884 s, 433 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0676547 s, 60.5 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 21:30:22 (1787794222) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 21:30:43 (1787794243) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 21:31:11 (1787794271) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 21:31:35 (1787794295) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 21:32:00 (1787794320) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 21:32:07 (1787794327) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 58869 mirror split pid 58870 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58869 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58870 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 21:33:16 (1787794396) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 4398 fallocating to largest ever: 0x133be 000001 1787794398.877653 fallocate 0x00a4e5 thru 0x0133bd (0x8ed9 bytes) skipping zero size read 000003 1787794398.902786 mapwrite 0x023dad thru 0x029576 (0x57ca bytes) fallocating to largest ever: 0x31d00 000004 1787794399.053690 fallocate 0x026d7a thru 0x031cff (0xaf86 bytes) 000005 1787794399.075774 mapread 0x0037bb thru 0x006489 (0x2ccf bytes) 000006 1787794399.100316 read 0x00a474 thru 0x010664 (0x61f1 bytes) 000007 1787794399.104089 punch from 0x028d44 to 0x02afc0 (0x227d bytes) 000008 1787794399.113243 fallocate 0x01f546 thru 0x02530d (0x5dc8 bytes) truncating to largest ever: 0x3c37e 000009 1787794399.121301 trunc from 0x031d00 to 0x03c37d (0xa67e bytes) 000010 1787794399.143320 mapread 0x029782 thru 0x03331a (0x9b99 bytes) 000011 1787794399.176820 fallocate 0x00ae14 thru 0x01366c (0x8859 bytes) 000012 1787794399.184233 trunc from 0x03be9d to 0x03c37d (0x4e1 bytes) 000013 1787794399.190268 punch from 0x001ba6 to 0x003481 (0x18dc bytes) 000015 1787794399.199105 trunc from 0x004b36 to 0x010c04 (0xc0cf bytes) 000016 1787794399.206687 trunc from 0x02e9e2 to 0x03b9ba (0xcfd9 bytes) 000019 1787794399.215669 punch from 0x00ea9b to 0x017d8d (0x92f3 bytes) 000020 1787794399.229458 MIRROR_EXTEND to 2 mirrors 000021 1787794400.042892 write 0x0320f0 thru 0x0340ff (0x2010 bytes) 000022 1787794400.092182 mapwrite 0x03315c thru 0x03bbfc (0x8aa1 bytes) fallocating to largest ever: 0x40000 000023 1787794400.148137 fallocate 0x03953d thru 0x03ffff (0x6ac3 bytes) 000024 1787794400.170945 mapwrite 0x03a8b1 thru 0x03ffff (0x574f bytes) 000025 1787794400.266048 read 0x003d3e thru 0x0071c3 (0x3486 bytes) 000028 1787794400.285067 MIRROR_RESYNC 2 mirrors 000029 1787794400.925605 mapwrite 0x039af0 thru 0x03ffff (0x6510 bytes) 000030 1787794401.061162 read 0x009695 thru 0x014c24 (0xb590 bytes) 000032 1787794401.077526 trunc from 0x019531 to 0x028769 (0xf239 bytes) 000033 1787794401.086988 read 0x027a09 thru 0x03077d (0x8d75 bytes) 000034 1787794401.123996 trunc from 0x040000 to 0x01b81b (0x247e4 bytes) 000035 1787794401.187682 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000036 1787794401.407162 mapread 0x004fc8 thru 0x00dc7b (0x8cb4 bytes) 000037 1787794401.465075 read 0x00aabe thru 0x016945 (0xbe88 bytes) 000039 1787794401.493575 trunc from 0x002348 to 0x00f427 (0xd0e0 bytes) 000040 1787794401.505409 write 0x015cd8 thru 0x017404 (0x172d bytes) 000041 1787794401.516164 punch from 0x001e2e to 0x00e1b7 (0xc38a bytes) 000042 1787794401.523447 write 0x01ae09 thru 0x024d9a (0x9f92 bytes) 000043 1787794401.530570 punch from 0x015486 to 0x015f70 (0xaeb bytes) 000044 1787794401.545271 trunc from 0x0225d6 to 0x024d9a (0x27c5 bytes) 000045 1787794401.559810 MIRROR_EXTEND to 2 mirrors 000046 1787794402.402318 mapwrite 0x00dcc8 thru 0x0144cc (0x6805 bytes) 000048 1787794402.518211 mapwrite 0x00648d thru 0x010be5 (0xa759 bytes) 000049 1787794402.554109 trunc from 0x024d9b to 0x00307b (0x21d1f bytes) 000050 1787794402.583688 write 0x02592a thru 0x030955 (0xb02c bytes) 000051 1787794402.586728 read 0x02df30 thru 0x030955 (0x2a26 bytes) 000052 1787794402.592120 trunc from 0x023a6c to 0x030955 (0xceea bytes) 000053 1787794402.619834 mapread 0x00644e thru 0x013f9c (0xdb4f bytes) 000055 1787794402.648761 mapwrite 0x02926b thru 0x034be2 (0xb978 bytes) 000056 1787794402.738803 trunc from 0x004b11 to 0x00ba02 (0x6ef2 bytes) 000057 1787794402.750165 trunc from 0x034be3 to 0x01ed32 (0x15eb0 bytes) 000058 1787794402.772795 write 0x01c8a2 thru 0x023ae4 (0x7243 bytes) 000059 1787794402.779345 punch from 0x0005e9 to 0x002dee (0x2806 bytes) 000060 1787794402.785468 trunc from 0x023ae5 to 0x014553 (0xf591 bytes) 000061 1787794402.819934 write 0x00c2f0 thru 0x00c7d0 (0x4e1 bytes) 000062 1787794402.821156 mapwrite 0x00225c thru 0x0029ef (0x794 bytes) 000063 1787794402.878052 punch from 0x005f3e to 0x010f03 (0xafc6 bytes) 000064 1787794402.889422 write 0x008ec3 thru 0x00d853 (0x4991 bytes) 000065 1787794402.924254 fallocate 0x00f77c thru 0x017d17 (0x859c bytes) 000066 1787794402.937388 write 0x0039a4 thru 0x00b170 (0x77cd bytes) 000067 1787794402.942317 trunc from 0x017d18 to 0x008832 (0xf4e5 bytes) 000069 1787794402.980118 read 0x000e3b thru 0x006fd3 (0x6199 bytes) 000070 1787794402.987931 punch from 0x0037c6 to 0x00391c (0x157 bytes) 000071 1787794403.017735 mapwrite 0x02860d thru 0x034b4b (0xc53f bytes) 000072 1787794403.148269 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34b4c): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b4c): Mirror 2: 0x3b7bacb5 (stale) CRC-32 checksum value for chunk [0, 0x34b4c): Mirror 1: 0x56049827 Mirror 3: 0x56049827 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b4c: skipped 000073 1787794404.122044 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34b4c): Mirror 2: 0x3b7bacb5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34b4c): Mirror 2: 0x3b7bacb5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34b4c: skipped 000074 1787794404.582123 read 0x00b3ab thru 0x010663 (0x52b9 bytes) 000075 1787794404.596982 mapread 0x007740 thru 0x00aa1e (0x32df bytes) 000076 1787794404.639138 MIRROR_RESYNC 2 mirrors 000078 1787794405.493566 read 0x022b3c thru 0x030278 (0xd73d bytes) 000079 1787794405.509556 trunc from 0x034b4c to 0x007b3f (0x2d00c bytes) 000080 1787794405.594865 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x7b40): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7b40): Mirror 2: 0xc083b5f2 (stale) CRC-32 checksum value for chunk [0, 0x7b40): Mirror 1: 0xc083b5f2 Mirror 3: 0xc083b5f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7b40: skipped 000081 1787794406.753259 punch from 0x006f58 to 0x007b3f (0xbe8 bytes) 000082 1787794406.828645 MIRROR_RESYNC 3 mirrors 000083 1787794408.140979 fallocate 0x034413 thru 0x03ffff (0xbbed bytes) 000084 1787794408.232961 trunc from 0x007b40 to 0x026f1a (0x1f3db bytes) 000085 1787794408.275902 mapread 0x01d145 thru 0x02034f (0x320b bytes) 000087 1787794408.323174 trunc from 0x007a2d to 0x013234 (0xb808 bytes) 000088 1787794408.335394 mapwrite 0x0146e6 thru 0x0222e6 (0xdc01 bytes) 000089 1787794408.419929 fallocate 0x038ad9 thru 0x03ffff (0x7527 bytes) 000090 1787794408.436197 read 0x00d633 thru 0x012e5b (0x5829 bytes) 000091 1787794408.440332 mapwrite 0x017863 thru 0x0212c0 (0x9a5e bytes) 000092 1787794408.493754 punch from 0x000202 to 0x00c77d (0xc57c bytes) 000093 1787794408.511095 fallocate 0x025454 thru 0x031d55 (0xc902 bytes) 000094 1787794408.527077 trunc from 0x026f1b to 0x017740 (0xf7da bytes) 000095 1787794408.583742 read 0x0050bc thru 0x012b22 (0xda67 bytes) 000096 1787794408.605147 fallocate 0x01c7cb thru 0x023250 (0x6a86 bytes) 000097 1787794408.611905 mapread 0x00c916 thru 0x00e76a (0x1e55 bytes) 000098 1787794408.613564 trunc from 0x005d1e to 0x00d19c (0x747f bytes) 000100 1787794408.625434 mapwrite 0x0119a0 thru 0x018797 (0x6df8 bytes) 000101 1787794408.739944 write 0x0373c8 thru 0x03b6f8 (0x4331 bytes) 000102 1787794408.743164 punch from 0x00b2c5 to 0x01a4b5 (0xf1f1 bytes) 000103 1787794408.752761 fallocate 0x029967 thru 0x037d80 (0xe41a bytes) 000104 1787794408.764345 punch from 0x01de62 to 0x02c909 (0xeaa8 bytes) 000105 1787794408.771053 mapwrite 0x0086fa thru 0x00d469 (0x4d70 bytes) 000106 1787794408.894284 trunc from 0x03b6f9 to 0x00fd5c (0x2b99c bytes) 000107 1787794408.923580 trunc from 0x007d2c to 0x008fdc (0x12b1 bytes) 000108 1787794408.931281 write 0x001634 thru 0x010381 (0xed4e bytes) 000110 1787794408.934079 read 0x006cda thru 0x00cfdd (0x6304 bytes) 000111 1787794408.935831 write 0x01afaa thru 0x01e589 (0x35e0 bytes) 000112 1787794408.943623 fallocate 0x03209a thru 0x037539 (0x54a0 bytes) 000114 1787794408.958719 write 0x030450 thru 0x038bbb (0x876c bytes) 000116 1787794408.961385 read 0x010b96 thru 0x01b418 (0xa883 bytes) 000117 1787794408.975422 punch from 0x023b2f to 0x0331cf (0xf6a1 bytes) 000118 1787794409.022692 trunc from 0x00f65f to 0x0124a8 (0x2e4a bytes) 000119 1787794409.038161 trunc from 0x0108d8 to 0x01dfd4 (0xd6fd bytes) 000120 1787794409.052101 trunc from 0x01d41b to 0x026813 (0x93f9 bytes) 000121 1787794409.086646 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38bbc): Mirror 2: 0xbc9f69a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bbc): Mirror 2: 0xbc9f69a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bbc: skipped 000122 1787794409.544351 mapread 0x005008 thru 0x012678 (0xd671 bytes) 000123 1787794409.591516 fallocate 0x028042 thru 0x032157 (0xa116 bytes) 000124 1787794409.597706 write 0x0348b9 thru 0x034eec (0x634 bytes) 000126 1787794409.633126 MIRROR_RESYNC 2 mirrors 000127 1787794410.848016 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000128 1787794410.992299 MIRROR_EXTEND to 2 mirrors 000129 1787794412.076431 read 0x008faa thru 0x018c05 (0xfc5c bytes) 000130 1787794412.095374 trunc from 0x0278ca to 0x031dd0 (0xa507 bytes) 000131 1787794412.147105 mapread 0x024ddd thru 0x027c1c (0x2e40 bytes) 000132 1787794412.177031 fallocate 0x004514 thru 0x00e289 (0x9d76 bytes) 000134 1787794412.184922 punch from 0x01802d to 0x021692 (0x9666 bytes) 000135 1787794412.195060 read 0x002ac9 thru 0x00ab12 (0x804a bytes) 000137 1787794412.212336 mapwrite 0x0053a3 thru 0x011ea5 (0xcb03 bytes) 000138 1787794412.282407 trunc from 0x0303b7 to 0x038bbb (0x8805 bytes) 000139 1787794412.309874 trunc from 0x038bbc to 0x02d083 (0xbb38 bytes) 000140 1787794412.389439 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000141 1787794412.598595 mapwrite 0x02085c thru 0x02fb75 (0xf31a bytes) 000142 1787794412.746527 trunc from 0x007320 to 0x01664d (0xf32e bytes) 000143 1787794412.757047 trunc from 0x02fb76 to 0x0275c7 (0x85ae bytes) 000144 1787794412.807571 mapwrite 0x030a0c thru 0x039bc4 (0x91b9 bytes) 000145 1787794412.929265 read 0x001d95 thru 0x001df9 (0x65 bytes) 000146 1787794412.949396 trunc from 0x011c7d to 0x014a73 (0x2df7 bytes) 000150 1787794412.975683 trunc from 0x025989 to 0x02f3b6 (0x9a2e bytes) 000151 1787794412.996556 write 0x036995 thru 0x03ffff (0x966b bytes) 000152 1787794413.000124 read 0x00b54e thru 0x00f608 (0x40bb bytes) 000153 1787794413.041455 trunc from 0x01368c to 0x014b14 (0x1489 bytes) 000154 1787794413.062753 mapread 0x008004 thru 0x013bb7 (0xbbb4 bytes) 000155 1787794413.078693 punch from 0x02a21e to 0x02afb6 (0xd99 bytes) 000156 1787794413.098219 punch from 0x02fd30 to 0x0307f4 (0xac5 bytes) 000157 1787794413.116709 mapread 0x033d8a thru 0x0376b4 (0x392b bytes) 000158 1787794413.118515 trunc from 0x00d63c to 0x01d50d (0xfed2 bytes) 000159 1787794413.126699 trunc from 0x02727a to 0x03246b (0xb1f2 bytes) 000160 1787794413.133303 read 0x016bc5 thru 0x01ba81 (0x4ebd bytes) 000161 1787794413.145665 mapwrite 0x00e620 thru 0x01ddd5 (0xf7b6 bytes) 000164 1787794413.173751 punch from 0x03a24c to 0x03f298 (0x504d bytes) 000165 1787794413.180957 trunc from 0x0057f4 to 0x0101ff (0xaa0c bytes) 000166 1787794413.187190 read 0x01bc21 thru 0x024c3a (0x901a bytes) 000167 1787794413.188174 punch from 0x03438c to 0x03ffff (0xbc74 bytes) 000169 1787794413.196415 mapwrite 0x02bf70 thru 0x03b24b (0xf2dc bytes) 000170 1787794413.282439 mapread 0x0201fc thru 0x020e6f (0xc74 bytes) 000171 1787794413.286656 trunc from 0x040000 to 0x01906b (0x26f94 bytes) 000173 1787794413.339856 mapwrite 0x0160cc thru 0x01cff8 (0x6f2d bytes) 000174 1787794413.446200 trunc from 0x01cff9 to 0x0335d7 (0x165df bytes) 000175 1787794413.485838 trunc from 0x0335d8 to 0x00e8ae (0x24d29 bytes) 000176 1787794413.522194 write 0x03194d thru 0x033983 (0x2037 bytes) 000177 1787794413.535726 MIRROR_EXTEND to 2 mirrors 000178 1787794414.323467 trunc from 0x033984 to 0x01ad85 (0x18bfe bytes) 000179 1787794414.436813 trunc from 0x01384d to 0x01ad85 (0x7539 bytes) 000180 1787794414.447109 punch from 0x0020a0 to 0x00e42c (0xc38d bytes) 000181 1787794414.463129 mapwrite 0x0315bd thru 0x03fc91 (0xe6d5 bytes) 000182 1787794414.662236 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fc92): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc92): Mirror 2: 0xc0ced09d (stale) CRC-32 checksum value for chunk [0, 0x3fc92): Mirror 1: 0x6505d8a8 Mirror 3: 0x6505d8a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc92: skipped 000183 1787794416.038438 punch from 0x033baa to 0x0341de (0x635 bytes) 000184 1787794416.088439 punch from 0x0146a5 to 0x0225a7 (0xdf03 bytes) 000185 1787794416.095929 trunc from 0x021799 to 0x0255b3 (0x3e1b bytes) 000186 1787794416.101771 trunc from 0x03fc92 to 0x012411 (0x2d880 bytes) 000187 1787794416.128954 trunc from 0x0079fa to 0x012411 (0xaa18 bytes) 000188 1787794416.135197 write 0x01cb05 thru 0x028a4a (0xbf46 bytes) 000189 1787794416.139281 fallocate 0x0077e0 thru 0x01561e (0xde3f bytes) 000190 1787794416.145638 fallocate 0x00c7ff thru 0x01c7e5 (0xffe7 bytes) 000191 1787794416.166294 MIRROR_RESYNC 3 mirrors 000192 1787794417.575393 trunc from 0x028a4b to 0x01ed58 (0x9cf2 bytes) 000193 1787794417.652559 trunc from 0x012b91 to 0x014f87 (0x23f7 bytes) 000194 1787794417.659958 trunc from 0x01ed59 to 0x01fb0c (0xdb4 bytes) 000195 1787794417.720730 trunc from 0x00beca to 0x0148ae (0x89e5 bytes) 000196 1787794417.738619 write 0x00ff54 thru 0x01c459 (0xc506 bytes) 000198 1787794417.773166 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1fb0d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fb0d): Mirror 2: 0xaefb9093 (stale) Mirror 3: 0xaefb9093 (stale) CRC-32 checksum value for chunk [0, 0x1fb0d): Mirror 1: 0xdf26632e Mirror 4: 0xdf26632e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fb0d: skipped 000199 1787794418.947967 trunc from 0x01fb0d to 0x02c81a (0xcd0e bytes) 000200 1787794419.036304 write 0x027eff thru 0x02fce8 (0x7dea bytes) 000201 1787794419.051514 trunc from 0x0002ad to 0x007440 (0x7194 bytes) 000202 1787794419.058117 fallocate 0x001e4c thru 0x00b0f2 (0x92a7 bytes) 000203 1787794419.065161 read 0x024ec9 thru 0x02fce8 (0xae20 bytes) 000204 1787794419.079262 read 0x01f77c thru 0x02a2ef (0xab74 bytes) 000205 1787794419.115268 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fce9): Mirror 3: 0x66b14b10 (stale) Mirror 4: 0x54999db2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fce9): Mirror 3: 0x66b14b10 (stale) Mirror 4: 0x54999db2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fce9: skipped 000206 1787794419.503423 mapwrite 0x00a6ae thru 0x014148 (0x9a9b bytes) 000207 1787794419.551184 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2fce9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fce9): Mirror 3: 0x66b14b10 (stale) Mirror 4: 0x54999db2 (stale) CRC-32 checksum value for chunk [0, 0x2fce9): Mirror 1: 0x977a5bad Mirror 5: 0x977a5bad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fce9: skipped 000208 1787794420.690009 trunc from 0x003695 to 0x005f92 (0x28fe bytes) 000209 1787794420.772034 read 0x011663 thru 0x015745 (0x40e3 bytes) 000210 1787794420.788889 fallocate 0x023daa thru 0x03210f (0xe366 bytes) 000211 1787794420.808375 MIRROR_RESYNC 4 mirrors 000212 1787794422.329314 fallocate 0x03f6e8 thru 0x03ffff (0x918 bytes) 000213 1787794422.378429 MIRROR_RESYNC 4 mirrors 000214 1787794423.938009 MIRROR_EXTEND to 5 mirrors 000215 1787794424.823789 fallocate 0x00f5ac thru 0x01d4f7 (0xdf4c bytes) 000216 1787794424.895656 punch from 0x025510 to 0x03382a (0xe31b bytes) 000217 1787794424.907744 read 0x0013bd thru 0x008800 (0x7444 bytes) 000218 1787794424.948991 mapwrite 0x03e7f7 thru 0x03ffff (0x1809 bytes) 000219 1787794424.990065 read 0x02d9b9 thru 0x039078 (0xb6c0 bytes) 000220 1787794425.004417 trunc from 0x040000 to 0x01bbeb (0x24414 bytes) 000221 1787794425.049253 trunc from 0x01bbec to 0x013be8 (0x8003 bytes) 000223 1787794425.075974 mapwrite 0x001194 thru 0x007b6f (0x69dc bytes) 000224 1787794425.101774 trunc from 0x00a29e to 0x010d09 (0x6a6c bytes) 000226 1787794425.114041 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13be9): Mirror 3: 0x72a811b7 (stale) Mirror 4: 0x72a811b7 (stale) Mirror 6: 0x72a811b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13be9): Mirror 3: 0x72a811b7 (stale) Mirror 4: 0x72a811b7 (stale) Mirror 6: 0x72a811b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13be9: skipped 000227 1787794425.514238 read 0x005ac2 thru 0x00baff (0x603e bytes) 000228 1787794425.533451 fallocate 0x039283 thru 0x03a90d (0x168b bytes) 000229 1787794425.543368 mapread 0x00ee47 thru 0x013be8 (0x4da2 bytes) 000230 1787794425.605303 mapwrite 0x037876 thru 0x03f8a5 (0x8030 bytes) 000232 1787794425.716994 read 0x015c76 thru 0x01f564 (0x98ef bytes) 000233 1787794425.730806 read 0x01342d thru 0x0223b8 (0xef8c bytes) 000234 1787794425.732125 punch from 0x03ca89 to 0x03f8a5 (0x2e1d bytes) 000235 1787794425.738871 punch from 0x02b7f8 to 0x03a931 (0xf13a bytes) 000237 1787794425.758033 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f8a6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8a6): Mirror 3: 0x9830940e (stale) Mirror 4: 0x9830940e (stale) Mirror 6: 0x9830940e (stale) CRC-32 checksum value for chunk [0, 0x3f8a6): Mirror 1: 0x9fc4d775 Mirror 7: 0x9fc4d775 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8a6: skipped 000238 1787794427.217526 write 0x006261 thru 0x00f1ab (0x8f4b bytes) 000239 1787794427.275735 mapwrite 0x01d704 thru 0x0207bd (0x30ba bytes) 000240 1787794427.335573 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f8a6): Mirror 3: 0x9830940e (stale) Mirror 4: 0x9830940e (stale) Mirror 6: 0x9830940e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8a6): Mirror 3: 0x9830940e (stale) Mirror 4: 0x9830940e (stale) Mirror 6: 0x9830940e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8a6: skipped 000241 1787794427.708858 write 0x01334d thru 0x018621 (0x52d5 bytes) 000242 1787794427.730229 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f8a6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f8a6): Mirror 3: 0x9830940e (stale) Mirror 4: 0x9830940e (stale) Mirror 6: 0x9830940e (stale) CRC-32 checksum value for chunk [0, 0x3f8a6): Mirror 1: 0x4ee509e5 Mirror 7: 0x4ee509e5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f8a6: skipped 000243 1787794429.163561 mapwrite 0x03d64e thru 0x03ffff (0x29b2 bytes) 000244 1787794429.274317 trunc from 0x03c1b3 to 0x03ffff (0x3e4d bytes) 000245 1787794429.281572 read 0x011532 thru 0x015477 (0x3f46 bytes) 000246 1787794429.306395 trunc from 0x002a5f to 0x0106ef (0xdc91 bytes) 000247 1787794429.330685 mapwrite 0x01f3f7 thru 0x02525e (0x5e68 bytes) 000248 1787794429.454118 MIRROR_RESYNC 5 mirrors 000249 1787794431.752770 mapread 0x03e492 thru 0x03ffff (0x1b6e bytes) 000250 1787794431.784416 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000251 1787794432.287467 punch from 0x00903d to 0x010e01 (0x7dc5 bytes) 000253 1787794432.344969 mapwrite 0x011679 thru 0x0139c1 (0x2349 bytes) 000254 1787794432.401849 trunc from 0x040000 to 0x00c040 (0x33fbf bytes) 000255 1787794432.444929 trunc from 0x00b350 to 0x00c040 (0xcf1 bytes) 000256 1787794432.459429 mapwrite 0x024363 thru 0x032636 (0xe2d4 bytes) 000258 1787794432.532030 mapread 0x025f22 thru 0x02bee7 (0x5fc6 bytes) 000260 1787794432.535444 read 0x00469d thru 0x008aa2 (0x4406 bytes) 000261 1787794432.552846 fallocate 0x014ce7 thru 0x01eeba (0xa1d4 bytes) 000262 1787794432.558438 read 0x026706 thru 0x030250 (0x9b4b bytes) 000264 1787794432.563912 read 0x00db4b thru 0x018f4c (0xb402 bytes) 000265 1787794432.577773 read 0x013911 thru 0x01e629 (0xad19 bytes) 000266 1787794432.578509 punch from 0x0294d1 to 0x030d41 (0x7871 bytes) 000267 1787794432.586967 trunc from 0x026ce5 to 0x032636 (0xb952 bytes) 000268 1787794432.595104 trunc from 0x032637 to 0x02552b (0xd10b bytes) 000270 1787794432.625945 trunc from 0x02552c to 0x03c77a (0x1724f bytes) 000271 1787794432.647982 mapread 0x02bedc thru 0x038427 (0xc54c bytes) 000272 1787794432.672133 MIRROR_RESYNC 4 mirrors 000273 1787794433.917034 punch from 0x013578 to 0x01689b (0x3324 bytes) 000274 1787794433.986777 read 0x01eb54 thru 0x02386c (0x4d19 bytes) 000275 1787794433.998283 read 0x0267ea thru 0x03153b (0xad52 bytes) 000276 1787794434.030022 mapwrite 0x026307 thru 0x030be7 (0xa8e1 bytes) 000277 1787794434.060004 fallocate 0x00df65 thru 0x0165e1 (0x867d bytes) 000278 1787794434.067800 mapread 0x0253db thru 0x029d91 (0x49b7 bytes) 000279 1787794434.068811 read 0x034adc thru 0x03a9e3 (0x5f08 bytes) 000280 1787794434.082011 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c77b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c77b): Mirror 3: 0x7a706a84 (stale) Mirror 4: 0x7a706a84 (stale) Mirror 7: 0x7a706a84 (stale) CRC-32 checksum value for chunk [0, 0x3c77b): Mirror 1: 0x2aa8b6b5 Mirror 8: 0x2aa8b6b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c77b: skipped 000282 1787794435.201330 read 0x000270 thru 0x009dc4 (0x9b55 bytes) 000283 1787794435.225623 read 0x01c5d7 thru 0x02c418 (0xfe42 bytes) 000284 1787794435.239993 punch from 0x0215d3 to 0x027e66 (0x6894 bytes) 000285 1787794435.307259 fallocate 0x00351c thru 0x00c875 (0x935a bytes) 000286 1787794435.326662 mapwrite 0x008f49 thru 0x00e890 (0x5948 bytes) 000287 1787794435.479046 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c77b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c77b): Mirror 3: 0x7a706a84 (stale) Mirror 4: 0x7a706a84 (stale) Mirror 7: 0x7a706a84 (stale) Mirror 8: 0x2aa8b6b5 (stale) CRC-32 checksum value for chunk [0, 0x3c77b): Mirror 1: 0xcf31a38d Mirror 9: 0xcf31a38d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c77b: skipped 000288 1787794436.926727 trunc from 0x0248c5 to 0x02f47e (0xabba bytes) 000289 1787794437.000974 fallocate 0x035e79 thru 0x03ffff (0xa187 bytes) 000290 1787794437.012730 punch from 0x026d92 to 0x029c37 (0x2ea6 bytes) 000291 1787794437.025135 trunc from 0x02e5c9 to 0x034011 (0x5a49 bytes) 000292 1787794437.039558 mapwrite 0x014fec thru 0x01c782 (0x7797 bytes) 000293 1787794437.115589 MIRROR_RESYNC 6 mirrors 000294 1787794439.737036 write 0x00f708 thru 0x011267 (0x1b60 bytes) 000295 1787794439.829957 punch from 0x01ddb4 to 0x027de2 (0xa02f bytes) 000296 1787794439.849849 punch from 0x020b14 to 0x02c717 (0xbc04 bytes) 000297 1787794439.859158 write 0x00f6df thru 0x0162da (0x6bfc bytes) 000298 1787794439.867836 punch from 0x02f8b5 to 0x03706f (0x77bb bytes) 000299 1787794439.893423 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x89dab6f2 (stale) Mirror 4: 0x89dab6f2 (stale) Mirror 8: 0x89dab6f2 (stale) Mirror 9: 0x89dab6f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x89dab6f2 (stale) Mirror 4: 0x89dab6f2 (stale) Mirror 8: 0x89dab6f2 (stale) Mirror 9: 0x89dab6f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000300 1787794440.604757 trunc from 0x040000 to 0x0344af (0xbb50 bytes) 000301 1787794440.624635 read 0x0274e5 thru 0x02a868 (0x3384 bytes) 000302 1787794440.631684 write 0x00c0d4 thru 0x0146fa (0x8627 bytes) 000303 1787794440.643239 mapread 0x010efe thru 0x01c444 (0xb547 bytes) 000305 1787794440.677879 mapread 0x013ecb thru 0x02284a (0xe980 bytes) 000306 1787794440.691580 read 0x024369 thru 0x02e8d3 (0xa56b bytes) 000307 1787794440.701853 read 0x000617 thru 0x00b0eb (0xaad5 bytes) 000308 1787794440.702470 mapread 0x01cc91 thru 0x02a106 (0xd476 bytes) 000309 1787794440.703423 trunc from 0x0344b0 to 0x025b48 (0xe967 bytes) 000310 1787794440.722370 punch from 0x001c3e to 0x00fbcb (0xdf8e bytes) 000311 1787794440.738382 mapread 0x00e301 thru 0x01a3c4 (0xc0c4 bytes) 000312 1787794440.754967 trunc from 0x025b49 to 0x01590a (0x1023e bytes) 000313 1787794440.781624 fallocate 0x030e52 thru 0x03c8c8 (0xba77 bytes) 000315 1787794440.805586 mapread 0x034fb7 thru 0x03c8c8 (0x7912 bytes) 000316 1787794440.838662 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c8c9): Mirror 4: 0xc739cb17 (stale) Mirror 8: 0xc739cb17 (stale) Mirror 9: 0xc739cb17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c8c9): Mirror 4: 0xc739cb17 (stale) Mirror 8: 0xc739cb17 (stale) Mirror 9: 0xc739cb17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c8c9: skipped 000317 1787794441.434254 punch from 0x039d29 to 0x03c8c8 (0x2ba0 bytes) 000318 1787794441.444927 write 0x0040eb thru 0x0065a8 (0x24be bytes) 000319 1787794441.467703 punch from 0x008dbe to 0x01008e (0x72d1 bytes) 000320 1787794441.473817 trunc from 0x03c8c9 to 0x0015ef (0x3b2d9 bytes) 000321 1787794441.499136 trunc from 0x0015f0 to 0x02b756 (0x2a167 bytes) 000322 1787794441.540391 MIRROR_RESYNC 4 mirrors 000324 1787794442.684397 write 0x00801a thru 0x00d2ab (0x5292 bytes) 000325 1787794442.779201 mapread 0x0001f3 thru 0x0002b7 (0xc5 bytes) 000326 1787794442.800848 write 0x027b3d thru 0x02b16b (0x362f bytes) 000327 1787794442.828811 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b757): Mirror 4: 0xe387918d (stale) Mirror 9: 0xe387918d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b757): Mirror 4: 0xe387918d (stale) Mirror 9: 0xe387918d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b757: skipped 000328 1787794443.377863 mapread 0x0294d6 thru 0x02b756 (0x2281 bytes) 000329 1787794443.401325 fallocate 0x01a198 thru 0x01d101 (0x2f6a bytes) 000330 1787794443.407121 read 0x02698d thru 0x0287d4 (0x1e48 bytes) 000331 1787794443.407869 mapread 0x00e71c thru 0x017f39 (0x981e bytes) 000332 1787794443.439203 trunc from 0x01d585 to 0x021408 (0x3e84 bytes) 000333 1787794443.448472 punch from 0x0188e6 to 0x01ebe9 (0x6304 bytes) 000334 1787794443.454998 read 0x02899a thru 0x02b756 (0x2dbd bytes) 000335 1787794443.455804 mapwrite 0x023430 thru 0x02a54b (0x711c bytes) 000336 1787794443.505775 mapwrite 0x02a0ae thru 0x03127e (0x71d1 bytes) 000337 1787794443.607258 punch from 0x02ac39 to 0x02c73a (0x1b02 bytes) 000339 1787794443.616690 mapread 0x005caa thru 0x00aa91 (0x4de8 bytes) 000340 1787794443.618123 punch from 0x01804a to 0x021132 (0x90e9 bytes) 000341 1787794443.634017 trunc from 0x03127f to 0x00d1e7 (0x24097 bytes) 000342 1787794443.657424 punch from 0x001256 to 0x00b222 (0x9fcd bytes) 000343 1787794443.664276 write 0x00bcc2 thru 0x00c934 (0xc73 bytes) 000344 1787794443.672126 mapread 0x002c82 thru 0x00d1e7 (0xa566 bytes) 000345 1787794443.730090 mapwrite 0x0197c4 thru 0x0255e5 (0xbe22 bytes) 000346 1787794443.899725 trunc from 0x0255e6 to 0x0387b8 (0x131d3 bytes) 000347 1787794443.951396 trunc from 0x0387b9 to 0x021464 (0x17354 bytes) 000348 1787794443.975819 mapread 0x017545 thru 0x01c159 (0x4c15 bytes) 000349 1787794443.987173 fallocate 0x004657 thru 0x01312b (0xead5 bytes) 000350 1787794443.991608 mapwrite 0x0309e8 thru 0x038700 (0x7d19 bytes) 000351 1787794444.064002 trunc from 0x038701 to 0x02e55b (0xa1a5 bytes) 000352 1787794444.084334 fallocate 0x023859 thru 0x02467b (0xe23 bytes) 000353 1787794444.098676 fallocate 0x03ca62 thru 0x03ffff (0x359e bytes) 000354 1787794444.114370 mapwrite 0x002653 thru 0x011531 (0xeedf bytes) 000355 1787794444.192382 mapwrite 0x02d2ad thru 0x03775e (0xa4b2 bytes) 000357 1787794444.321386 punch from 0x035d0a to 0x03ffff (0xa2f6 bytes) 000358 1787794444.331983 MIRROR_RESYNC 3 mirrors 000359 1787794445.609512 trunc from 0x040000 to 0x03fa80 (0x57f bytes) 000360 1787794445.647828 fallocate 0x038f33 thru 0x03ffff (0x70cd bytes) 000361 1787794445.655740 mapread 0x028c65 thru 0x02f4fe (0x689a bytes) 000362 1787794445.688695 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fa81): Mirror 4: 0x3901a6a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa81): Mirror 4: 0x3901a6a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa81: skipped 000363 1787794446.071040 mapwrite 0x0182b7 thru 0x01f86d (0x75b7 bytes) 000364 1787794446.143325 trunc from 0x03fa81 to 0x02f296 (0x107ea bytes) 000365 1787794446.173610 trunc from 0x007d67 to 0x00a966 (0x2c00 bytes) 000366 1787794446.181999 punch from 0x00676e to 0x011bb2 (0xb445 bytes) 000367 1787794446.195333 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f297): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f297): Mirror 4: 0x1ecc50b4 (stale) CRC-32 checksum value for chunk [0, 0x2f297): Mirror 1: 0xaf24d1f8 Mirror 5: 0xaf24d1f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f297: skipped 000369 1787794447.343792 punch from 0x026f59 to 0x02a2f1 (0x3399 bytes) 000370 1787794447.386359 fallocate 0x030709 thru 0x03ab4d (0xa445 bytes) 000371 1787794447.393608 mapread 0x00273c thru 0x009013 (0x68d8 bytes) 000372 1787794447.413698 mapwrite 0x005c4f thru 0x00a90f (0x4cc1 bytes) 000373 1787794447.437427 trunc from 0x01b464 to 0x01d069 (0x1c06 bytes) 000374 1787794447.444375 punch from 0x006b8b to 0x013f02 (0xd378 bytes) 000375 1787794447.452701 trunc from 0x03ab4e to 0x038a1e (0x212f bytes) 000376 1787794447.490333 fallocate 0x03ef00 thru 0x03ffff (0x1100 bytes) 000377 1787794447.497721 trunc from 0x008a74 to 0x00966c (0xbf9 bytes) 000378 1787794447.505970 mapwrite 0x024c50 thru 0x031186 (0xc537 bytes) 000379 1787794447.577163 mapwrite 0x00a432 thru 0x0175c6 (0xd195 bytes) 000380 1787794447.636528 punch from 0x01344d to 0x01db12 (0xa6c6 bytes) 000381 1787794447.649256 mapread 0x031b9b thru 0x037fe4 (0x644a bytes) 000382 1787794447.677756 write 0x012663 thru 0x014c64 (0x2602 bytes) 000383 1787794447.684418 mapwrite 0x00b6d2 thru 0x01a907 (0xf236 bytes) 000384 1787794447.735036 mapread 0x031457 thru 0x03afe8 (0x9b92 bytes) 000385 1787794447.736113 fallocate 0x00a2e2 thru 0x01193b (0x765a bytes) 000386 1787794447.744464 mapwrite 0x00affe thru 0x00e06a (0x306d bytes) 000387 1787794447.773643 write 0x03ba79 thru 0x03ffff (0x4587 bytes) 000388 1787794447.774891 read 0x017e8d thru 0x022601 (0xa775 bytes) 000389 1787794447.777024 write 0x0156a0 thru 0x017a73 (0x23d4 bytes) 000390 1787794447.784628 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa06dbebf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa06dbebf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000392 1787794448.149269 mapread 0x01a89c thru 0x01e283 (0x39e8 bytes) 000393 1787794448.175045 fallocate 0x021598 thru 0x02dcfe (0xc767 bytes) 000394 1787794448.184401 mapwrite 0x0206fd thru 0x024a8f (0x4393 bytes) 000395 1787794448.213262 read 0x03ea32 thru 0x03ff8d (0x155c bytes) 000396 1787794448.224339 punch from 0x028761 to 0x0383d4 (0xfc74 bytes) 000397 1787794448.235142 mapread 0x014ec5 thru 0x01aab6 (0x5bf2 bytes) 000398 1787794448.255130 write 0x03e8b2 thru 0x03ffff (0x174e bytes) 000399 1787794448.264313 mapwrite 0x009478 thru 0x011b42 (0x86cb bytes) 000400 1787794448.306926 mapwrite 0x01acde thru 0x01d1dc (0x24ff bytes) 000401 1787794448.332124 write 0x034b5f thru 0x03ffff (0xb4a1 bytes) 000403 1787794448.344133 mapwrite 0x02d890 thru 0x02e5f1 (0xd62 bytes) 000404 1787794448.392430 mapwrite 0x012add thru 0x015708 (0x2c2c bytes) 000405 1787794448.418520 trunc from 0x0359db to 0x03ffff (0xa625 bytes) 000406 1787794448.425204 mapread 0x001797 thru 0x001b49 (0x3b3 bytes) 000407 1787794448.433396 fallocate 0x029ecd thru 0x036a40 (0xcb74 bytes) 000408 1787794448.440038 mapwrite 0x0371c4 thru 0x03ffff (0x8e3c bytes) 000409 1787794448.481349 mapwrite 0x034bf4 thru 0x039237 (0x4644 bytes) 000410 1787794448.512327 fallocate 0x03e22d thru 0x03ffff (0x1dd3 bytes) 000412 1787794448.541709 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa06dbebf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x515722bc Mirror 5: 0x515722bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000413 1787794449.801706 punch from 0x022130 to 0x031e53 (0xfd24 bytes) 000414 1787794449.847957 punch from 0x03efec to 0x03ffff (0x1014 bytes) 000415 1787794449.857372 mapread 0x01edb6 thru 0x028e3b (0xa086 bytes) 000417 1787794449.880930 trunc from 0x040000 to 0x0049f9 (0x3b606 bytes) 000418 1787794449.919896 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x49fa): Mirror 5: 0x5304e03f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x49fa): Mirror 5: 0x5304e03f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x49fa: skipped 000419 1787794450.301076 mapread 0x0009c8 thru 0x0047f5 (0x3e2e bytes) 000420 1787794450.324641 trunc from 0x00332a to 0x0049f9 (0x16d0 bytes) 000421 1787794450.334932 trunc from 0x0049fa to 0x03e143 (0x3974a bytes) 000422 1787794450.379857 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e144): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e144): Mirror 5: 0xe5e01205 (stale) CRC-32 checksum value for chunk [0, 0x3e144): Mirror 1: 0x6a1c6e1 Mirror 6: 0x6a1c6e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e144: skipped 000423 1787794451.878404 fallocate 0x03e1e6 thru 0x03ffff (0x1e1a bytes) 000424 1787794451.943726 trunc from 0x038b6f to 0x039daf (0x1241 bytes) 000425 1787794451.953703 mapread 0x021e64 thru 0x024e75 (0x3012 bytes) 000426 1787794451.981820 write 0x037ab9 thru 0x03e75a (0x6ca2 bytes) 000427 1787794452.003058 trunc from 0x011e23 to 0x01a956 (0x8b34 bytes) 000428 1787794452.018033 write 0x03fa3b thru 0x03ffff (0x5c5 bytes) 000429 1787794452.037288 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x515722bc (stale) Mirror 6: 0x40c759c4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x83964813 Mirror 7: 0x83964813 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000430 1787794453.304917 trunc from 0x040000 to 0x0334ad (0xcb52 bytes) 000431 1787794453.383217 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x334ae): Mirror 6: 0xb42d26dd (stale) Mirror 7: 0xb42d26dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x334ae): Mirror 6: 0xb42d26dd (stale) Mirror 7: 0xb42d26dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x334ae: skipped 000432 1787794453.780442 write 0x036e88 thru 0x03ffff (0x9178 bytes) 000433 1787794453.790568 write 0x01407b thru 0x01a647 (0x65cd bytes) 000434 1787794453.802747 mapread 0x037d01 thru 0x03ffff (0x82ff bytes) 000435 1787794453.806213 trunc from 0x011771 to 0x01f025 (0xd8b5 bytes) 000436 1787794453.854763 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x83964813 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x83964813 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000437 1787794454.313328 mapwrite 0x019505 thru 0x028b33 (0xf62f bytes) 000438 1787794454.433795 fallocate 0x01f418 thru 0x0250bf (0x5ca8 bytes) 000439 1787794454.452270 fallocate 0x013155 thru 0x01a71b (0x75c7 bytes) 000440 1787794454.483337 punch from 0x00380a to 0x013178 (0xf96f bytes) 000441 1787794454.504091 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x83964813 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x14c03998 Mirror 8: 0x14c03998 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000442 1787794455.758547 mapread 0x008288 thru 0x011b31 (0x98aa bytes) 000443 1787794455.798369 trunc from 0x0093d2 to 0x0155de (0xc20d bytes) 000444 1787794455.875440 read 0x038d46 thru 0x03ffff (0x72ba bytes) 000445 1787794455.901858 mapwrite 0x00159d thru 0x00a84d (0x92b1 bytes) 000446 1787794455.963467 trunc from 0x001236 to 0x003f44 (0x2d0f bytes) 000447 1787794455.979577 trunc from 0x040000 to 0x01694b (0x296b4 bytes) 000448 1787794456.041563 read 0x011d7f thru 0x01694b (0x4bcd bytes) 000450 1787794456.061096 fallocate 0x024afd thru 0x0261c4 (0x16c8 bytes) 000451 1787794456.069101 mapread 0x00f7d0 thru 0x012ad7 (0x3308 bytes) 000452 1787794456.074881 mapwrite 0x0395f9 thru 0x039b52 (0x55a bytes) 000453 1787794456.160537 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39b53): Mirror 8: 0xc7593d94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b53): Mirror 8: 0xc7593d94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b53: skipped 000454 1787794456.608869 read 0x01f5d4 thru 0x02b66b (0xc098 bytes) 000455 1787794456.623267 mapread 0x012cc7 thru 0x01e6d3 (0xba0d bytes) 000456 1787794456.658428 read 0x023f42 thru 0x02cc85 (0x8d44 bytes) 000457 1787794456.683195 write 0x039f76 thru 0x03ffff (0x608a bytes) 000458 1787794456.691880 trunc from 0x040000 to 0x02b888 (0x14777 bytes) 000460 1787794456.717143 mapwrite 0x026b62 thru 0x0366b7 (0xfb56 bytes) 000461 1787794456.844070 fallocate 0x004a43 thru 0x00f65a (0xac18 bytes) 000462 1787794456.852000 mapwrite 0x0266fc thru 0x0319fa (0xb2ff bytes) 000463 1787794456.876238 mapwrite 0x01c66c thru 0x029444 (0xcdd9 bytes) 000464 1787794456.922260 trunc from 0x0366b8 to 0x018372 (0x1e345 bytes) 000465 1787794456.959165 trunc from 0x00e27a to 0x018372 (0xa0f9 bytes) 000466 1787794456.973230 mapread 0x009e65 thru 0x0176f0 (0xd88c bytes) 000467 1787794457.045807 mapwrite 0x0027d5 thru 0x003563 (0xd8f bytes) 000468 1787794457.061755 mapwrite 0x012358 thru 0x020fe5 (0xec8e bytes) 000469 1787794457.170473 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20fe6): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20fe6): Mirror 8: 0x9d8c40a (stale) CRC-32 checksum value for chunk [0, 0x20fe6): Mirror 1: 0x92778eed Mirror 9: 0x92778eed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20fe6: skipped 000470 1787794458.545304 read 0x019f6d thru 0x01a1c2 (0x256 bytes) 000471 1787794458.555976 MIRROR_RESYNC 3 mirrors 000472 1787794459.573715 fallocate 0x032695 thru 0x03bfc7 (0x9933 bytes) 000473 1787794459.653811 punch from 0x01f0db to 0x020fe5 (0x1f0b bytes) 000474 1787794459.678744 mapread 0x00f91b thru 0x0110df (0x17c5 bytes) 000475 1787794459.723422 fallocate 0x026296 thru 0x02a831 (0x459c bytes) 000476 1787794459.730791 fallocate 0x007e7b thru 0x00bc4d (0x3dd3 bytes) 000477 1787794459.743025 fallocate 0x01a07b thru 0x0235c8 (0x954e bytes) 000478 1787794459.764036 punch from 0x013f8e to 0x01efe5 (0xb058 bytes) 000479 1787794459.778091 punch from 0x01cd29 to 0x01d7a2 (0xa7a bytes) 000480 1787794459.795168 trunc from 0x020fe6 to 0x030af2 (0xfb0d bytes) 000481 1787794459.826097 punch from 0x011dbe to 0x021919 (0xfb5c bytes) 000482 1787794459.832306 mapread 0x00e263 thru 0x011df9 (0x3b97 bytes) 000483 1787794459.844732 fallocate 0x02df28 thru 0x033637 (0x5710 bytes) 000484 1787794459.853429 trunc from 0x028d76 to 0x02aa23 (0x1cae bytes) 000485 1787794459.859514 mapwrite 0x003fa6 thru 0x00403f (0x9a bytes) 000486 1787794459.893947 fallocate 0x034c23 thru 0x03ffff (0xb3dd bytes) 000487 1787794459.901470 mapwrite 0x01f7e1 thru 0x02673d (0x6f5d bytes) 000488 1787794459.944347 punch from 0x00c727 to 0x0177eb (0xb0c5 bytes) 000489 1787794459.965781 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x904090f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x904090f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000490 1787794460.265236 punch from 0x017e1f to 0x02374b (0xb92d bytes) 000491 1787794460.277342 mapread 0x0230ee thru 0x02fb4a (0xca5d bytes) 000492 1787794460.310815 read 0x03d641 thru 0x03ffff (0x29bf bytes) 000493 1787794460.318329 trunc from 0x038b05 to 0x03ffff (0x74fb bytes) 000494 1787794460.329441 MIRROR_RESYNC 2 mirrors 000495 1787794461.249447 mapwrite 0x016f2e thru 0x017b8b (0xc5e bytes) 000496 1787794461.390229 write 0x00bd5f thru 0x014dab (0x904d bytes) 000497 1787794461.399530 write 0x03a243 thru 0x03ffff (0x5dbd bytes) 000498 1787794461.420311 write 0x02f683 thru 0x037a56 (0x83d4 bytes) 000499 1787794461.458757 read 0x000421 thru 0x005d7a (0x595a bytes) 000500 1787794461.473151 write 0x02bd63 thru 0x034d8c (0x902a bytes) 000501 1787794461.478916 write 0x0363b9 thru 0x03873b (0x2383 bytes) 000502 1787794461.484924 write 0x00d6b1 thru 0x01a61f (0xcf6f bytes) 000503 1787794461.492225 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000504 1787794461.708747 trunc from 0x040000 to 0x025290 (0x1ad6f bytes) 000505 1787794461.774466 mapread 0x015191 thru 0x023073 (0xdee3 bytes) 000506 1787794461.824024 trunc from 0x025291 to 0x0332a0 (0xe010 bytes) 000507 1787794461.838978 punch from 0x023f46 to 0x02cbb7 (0x8c72 bytes) 000508 1787794461.844435 fallocate 0x0340f5 thru 0x039dd0 (0x5cdc bytes) 000509 1787794461.851274 mapwrite 0x031cc8 thru 0x03fb83 (0xdebc bytes) 000510 1787794461.949056 mapread 0x019836 thru 0x0199e2 (0x1ad bytes) 000512 1787794461.952224 mapwrite 0x00c343 thru 0x010ddb (0x4a99 bytes) 000513 1787794461.983373 trunc from 0x002e5f to 0x012985 (0xfb27 bytes) 000514 1787794461.990244 trunc from 0x03fb84 to 0x019309 (0x2687a bytes) 000516 1787794462.016271 read 0x00cb8d thru 0x00f5e7 (0x2a5b bytes) 000517 1787794462.023462 fallocate 0x032647 thru 0x03dfb9 (0xb973 bytes) 000518 1787794462.030174 punch from 0x00adeb to 0x0159f5 (0xac0b bytes) 000519 1787794462.038223 write 0x010e45 thru 0x012027 (0x11e3 bytes) 000520 1787794462.067076 mapwrite 0x019ce8 thru 0x021170 (0x7489 bytes) 000521 1787794462.136433 fallocate 0x031c20 thru 0x033053 (0x1434 bytes) 000522 1787794462.158190 trunc from 0x03dfba to 0x022044 (0x1bf75 bytes) 000524 1787794462.200647 fallocate 0x01b81b thru 0x023d7e (0x8564 bytes) 000525 1787794462.213857 trunc from 0x022045 to 0x00725f (0x1ade5 bytes) 000526 1787794462.253028 trunc from 0x007260 to 0x02a331 (0x230d2 bytes) 000527 1787794462.291853 punch from 0x01319e to 0x013f41 (0xda4 bytes) 000528 1787794462.309038 fallocate 0x012ac7 thru 0x013c2a (0x1164 bytes) 000529 1787794462.320375 trunc from 0x00ddc9 to 0x01b565 (0xd79d bytes) 000530 1787794462.332841 punch from 0x022e73 to 0x02a331 (0x74bf bytes) 000532 1787794462.338187 punch from 0x01700f to 0x02451a (0xd50c bytes) 000533 1787794462.347871 mapread 0x0151c8 thru 0x021034 (0xbe6d bytes) 000534 1787794462.387234 mapwrite 0x002edd thru 0x0073e5 (0x4509 bytes) 000535 1787794462.433215 write 0x0236f0 thru 0x024c0e (0x151f bytes) 000536 1787794462.447111 MIRROR_EXTEND to 2 mirrors 000537 1787794463.574968 punch from 0x0292a2 to 0x02a331 (0x1090 bytes) 000538 1787794463.623916 fallocate 0x032fe2 thru 0x03f790 (0xc7af bytes) 000539 1787794463.635170 trunc from 0x0106f3 to 0x01e34b (0xdc59 bytes) 000540 1787794463.648219 mapread 0x00492c thru 0x00b610 (0x6ce5 bytes) 000542 1787794463.688563 fallocate 0x030a22 thru 0x035553 (0x4b32 bytes) 000543 1787794463.700868 mapwrite 0x038aa1 thru 0x03f0be (0x661e bytes) 000544 1787794463.764138 fallocate 0x02d307 thru 0x033502 (0x61fc bytes) 000545 1787794463.770374 fallocate 0x03f369 thru 0x03ffff (0xc97 bytes) 000546 1787794463.777218 punch from 0x02f7d0 to 0x03f059 (0xf88a bytes) 000547 1787794463.786965 trunc from 0x03f0bf to 0x0049f5 (0x3a6c9 bytes) 000548 1787794463.822177 punch from 0x002120 to 0x003d11 (0x1bf2 bytes) 000549 1787794463.828452 fallocate 0x0119d4 thru 0x01b42e (0x9a5b bytes) 000550 1787794463.834925 mapwrite 0x008c74 thru 0x00e11c (0x54a9 bytes) 000551 1787794463.904215 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xe11d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe11d): Mirror 2: 0x236f60d2 (stale) CRC-32 checksum value for chunk [0, 0xe11d): Mirror 1: 0xd03e03ed Mirror 3: 0xd03e03ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe11d: skipped 000552 1787794464.650936 mapwrite 0x02b2e7 thru 0x02e773 (0x348d bytes) 000553 1787794464.838380 write 0x0194c8 thru 0x026b69 (0xd6a2 bytes) 000554 1787794464.852188 trunc from 0x00f718 to 0x00f8aa (0x193 bytes) 000555 1787794464.861872 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2e774): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e774): Mirror 2: 0x24ffe726 (stale) Mirror 3: 0xd1546534 (stale) CRC-32 checksum value for chunk [0, 0x2e774): Mirror 1: 0xbc873270 Mirror 4: 0xbc873270 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e774: skipped 000556 1787794465.974112 write 0x00dd34 thru 0x00e4c6 (0x793 bytes) 000557 1787794466.039037 fallocate 0x021d38 thru 0x02e59a (0xc863 bytes) 000559 1787794466.045374 mapread 0x0134ef thru 0x01873f (0x5251 bytes) 000560 1787794466.079116 write 0x01e789 thru 0x02773d (0x8fb5 bytes) 000561 1787794466.095895 mapwrite 0x01588c thru 0x01c978 (0x70ed bytes) 000562 1787794466.135330 read 0x020ce7 thru 0x02b252 (0xa56c bytes) 000563 1787794466.142407 read 0x00c7e5 thru 0x00d4fa (0xd16 bytes) 000564 1787794466.162841 fallocate 0x03c565 thru 0x03ffff (0x3a9b bytes) 000565 1787794466.171349 trunc from 0x03b3b0 to 0x03ffff (0x4c50 bytes) 000566 1787794466.184371 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3d8681f6 (stale) Mirror 3: 0xa301ca18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3d8681f6 (stale) Mirror 3: 0xa301ca18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000567 1787794466.555588 mapread 0x00f5d2 thru 0x01c415 (0xce44 bytes) 000568 1787794466.583897 punch from 0x0220a7 to 0x0250cf (0x3029 bytes) 000569 1787794466.596043 MIRROR_RESYNC 3 mirrors 000570 1787794467.915823 trunc from 0x040000 to 0x003a1b (0x3c5e4 bytes) 000571 1787794467.972805 read 0x000c58 thru 0x003a1b (0x2dc4 bytes) 000572 1787794467.983858 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a1c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1c): Mirror 2: 0x2e1015cd (stale) Mirror 3: 0x2e1015cd (stale) CRC-32 checksum value for chunk [0, 0x3a1c): Mirror 1: 0x2e1015cd Mirror 4: 0x2e1015cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1c: skipped 000573 1787794468.989366 trunc from 0x000d9e to 0x003527 (0x278a bytes) 000574 1787794469.040974 punch from 0x001973 to 0x003a1b (0x20a9 bytes) 000575 1787794469.047213 punch from 0x000a06 to 0x003a1b (0x3016 bytes) 000576 1787794469.055991 trunc from 0x003a1c to 0x01d841 (0x19e26 bytes) 000577 1787794469.089019 write 0x00d690 thru 0x01998d (0xc2fe bytes) 000578 1787794469.111274 write 0x01f22e thru 0x023143 (0x3f16 bytes) 000579 1787794469.113446 fallocate 0x00f577 thru 0x01883d (0x92c7 bytes) 000580 1787794469.125735 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23144): Mirror 2: 0x7672d1be (stale) Mirror 3: 0x7672d1be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23144): Mirror 2: 0x7672d1be (stale) Mirror 3: 0x7672d1be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23144: skipped 000581 1787794469.471169 mapread 0x019869 thru 0x023143 (0x98db bytes) 000582 1787794469.498254 read 0x016772 thru 0x023143 (0xc9d2 bytes) 000583 1787794469.499239 mapread 0x021e97 thru 0x023143 (0x12ad bytes) 000584 1787794469.500952 read 0x0218f4 thru 0x02291d (0x102a bytes) 000585 1787794469.502078 punch from 0x0143e8 to 0x015c48 (0x1861 bytes) 000586 1787794469.507164 fallocate 0x025761 thru 0x026101 (0x9a1 bytes) 000587 1787794469.513418 trunc from 0x00fe2e to 0x01f0a0 (0xf273 bytes) 000588 1787794469.519765 write 0x011bf7 thru 0x01df8c (0xc396 bytes) 000590 1787794469.535345 trunc from 0x026102 to 0x002bc3 (0x2353e bytes) 000591 1787794469.556328 mapread 0x001a8b thru 0x002bc3 (0x1139 bytes) 000592 1787794469.557470 fallocate 0x029528 thru 0x036366 (0xce3f bytes) 000593 1787794469.568199 trunc from 0x002bc4 to 0x00e4a3 (0xb8e0 bytes) 000594 1787794469.594314 mapread 0x001d8c thru 0x00cedc (0xb151 bytes) 000595 1787794469.646291 mapwrite 0x0253b2 thru 0x0330f2 (0xdd41 bytes) 000596 1787794469.709203 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x330f3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x330f3): Mirror 2: 0xc6003184 (stale) Mirror 3: 0xc6003184 (stale) CRC-32 checksum value for chunk [0, 0x330f3): Mirror 1: 0xdd053fb1 Mirror 4: 0xdd053fb1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x330f3: skipped 000597 1787794470.499307 fallocate 0x0382ac thru 0x03d03b (0x4d90 bytes) 000598 1787794470.548423 fallocate 0x02a6fb thru 0x031b3a (0x7440 bytes) 000599 1787794470.554128 trunc from 0x01bdf3 to 0x029830 (0xda3e bytes) 000600 1787794470.560268 mapwrite 0x00acc1 thru 0x01464e (0x998e bytes) 000601 1787794470.603304 punch from 0x0129a1 to 0x01efdd (0xc63d bytes) 000602 1787794470.608831 read 0x032384 thru 0x0330f2 (0xd6f bytes) 000603 1787794470.614237 trunc from 0x01baf0 to 0x01c0b2 (0x5c3 bytes) 000604 1787794470.619591 mapread 0x0272de thru 0x02f885 (0x85a8 bytes) 000605 1787794470.629813 mapwrite 0x019539 thru 0x02390e (0xa3d6 bytes) 000607 1787794470.663091 mapread 0x01cd39 thru 0x01d811 (0xad9 bytes) 000609 1787794470.664467 punch from 0x0066ed to 0x014b3b (0xe44f bytes) 000610 1787794470.670338 fallocate 0x02497c thru 0x029187 (0x480c bytes) 000611 1787794470.675733 write 0x0015b7 thru 0x009904 (0x834e bytes) 000612 1787794470.690075 mapread 0x001e25 thru 0x00d051 (0xb22d bytes) 000613 1787794470.702913 trunc from 0x0006e0 to 0x00713d (0x6a5e bytes) 000614 1787794470.722089 write 0x01a274 thru 0x0257bf (0xb54c bytes) 000615 1787794470.724750 mapwrite 0x00b06f thru 0x0124dd (0x746f bytes) 000616 1787794470.763962 trunc from 0x028f1b to 0x0324d1 (0x95b7 bytes) 000617 1787794470.780156 mapwrite 0x02dc9e thru 0x031cc0 (0x4023 bytes) 000618 1787794470.851830 trunc from 0x02928e to 0x0330f2 (0x9e65 bytes) 000620 1787794470.863699 mapread 0x00584f thru 0x00d50f (0x7cc1 bytes) 000621 1787794470.898785 MIRROR_RESYNC 4 mirrors 000623 1787794472.976779 read 0x0255fb thru 0x02cec0 (0x78c6 bytes) 000625 1787794472.987350 trunc from 0x004944 to 0x007105 (0x27c2 bytes) 000626 1787794473.035858 trunc from 0x01e50f to 0x02c6e2 (0xe1d4 bytes) 000627 1787794473.042839 write 0x0306e5 thru 0x03b9f2 (0xb30e bytes) 000628 1787794473.049933 write 0x035015 thru 0x039b54 (0x4b40 bytes) 000630 1787794473.053104 trunc from 0x029d68 to 0x032315 (0x85ae bytes) 000631 1787794473.076052 read 0x0127dd thru 0x019dc7 (0x75eb bytes) 000632 1787794473.094425 MIRROR_RESYNC 4 mirrors 000633 1787794474.742946 MIRROR_RESYNC 4 mirrors 000634 1787794475.267901 trunc from 0x00483d to 0x00b429 (0x6bed bytes) 000635 1787794475.340759 punch from 0x000b02 to 0x00a2f6 (0x97f5 bytes) 000636 1787794475.357315 MIRROR_RESYNC 4 mirrors 000637 1787794476.848039 punch from 0x024c4e to 0x02bf69 (0x731c bytes) 000638 1787794476.902991 write 0x003e4d thru 0x00a744 (0x68f8 bytes) 000639 1787794476.924861 fallocate 0x02b6ec thru 0x035f1a (0xa82f bytes) 000642 1787794476.951190 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b9f3): Mirror 3: 0xedad813c (stale) Mirror 4: 0xedad813c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b9f3): Mirror 3: 0xedad813c (stale) Mirror 4: 0xedad813c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b9f3: skipped 000643 1787794477.406762 MIRROR_RESYNC 3 mirrors 000644 1787794478.762715 trunc from 0x03b9f3 to 0x018a70 (0x22f82 bytes) 000645 1787794478.806729 punch from 0x00d95f to 0x014689 (0x6d2b bytes) 000646 1787794478.816434 fallocate 0x01bfb1 thru 0x022059 (0x60a9 bytes) 000648 1787794478.839794 punch from 0x01018a to 0x01e240 (0xe0b7 bytes) 000649 1787794478.854262 read 0x0177ed thru 0x01c419 (0x4c2d bytes) 000650 1787794478.876241 trunc from 0x02205a to 0x02704c (0x4ff3 bytes) 000651 1787794478.923171 punch from 0x00a1b9 to 0x016962 (0xc7aa bytes) 000652 1787794478.934279 trunc from 0x00f102 to 0x015ec7 (0x6dc6 bytes) 000653 1787794478.943152 trunc from 0x02704d to 0x00d6f2 (0x1995a bytes) 000655 1787794478.979730 write 0x03af6a thru 0x03ffff (0x5096 bytes) 000656 1787794478.984744 fallocate 0x01ec4c thru 0x027249 (0x85fe bytes) 000657 1787794478.991024 mapread 0x020b70 thru 0x02a8f9 (0x9d8a bytes) 000658 1787794479.032700 MIRROR_RESYNC 3 mirrors 000659 1787794480.028526 trunc from 0x034540 to 0x03c189 (0x7c4a bytes) 000660 1787794480.101097 trunc from 0x0109a5 to 0x011d26 (0x1382 bytes) 000661 1787794480.110929 write 0x00a4fe thru 0x00bbf7 (0x16fa bytes) 000662 1787794480.143225 trunc from 0x040000 to 0x016174 (0x29e8b bytes) 000663 1787794480.190806 write 0x0292ea thru 0x02a7a7 (0x14be bytes) 000664 1787794480.202271 fallocate 0x0367d0 thru 0x03ffff (0x9830 bytes) 000665 1787794480.216580 trunc from 0x00b9e0 to 0x018b11 (0xd132 bytes) 000666 1787794480.254053 trunc from 0x02a7a8 to 0x034dfb (0xa654 bytes) 000667 1787794480.278354 read 0x008f30 thru 0x0108a6 (0x7977 bytes) 000669 1787794480.301323 write 0x032802 thru 0x03e9bf (0xc1be bytes) 000670 1787794480.312142 fallocate 0x000bc8 thru 0x002c8c (0x20c5 bytes) 000671 1787794480.318623 fallocate 0x012201 thru 0x01a77f (0x857f bytes) 000672 1787794480.333790 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e9c0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9c0): Mirror 3: 0xddbd27c9 (stale) Mirror 4: 0xddbd27c9 (stale) CRC-32 checksum value for chunk [0, 0x3e9c0): Mirror 1: 0xa6fce6bf Mirror 5: 0xa6fce6bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9c0: skipped 000673 1787794481.506615 mapread 0x02aae5 thru 0x03a794 (0xfcb0 bytes) 000675 1787794481.590574 read 0x009b27 thru 0x0178c4 (0xdd9e bytes) 000677 1787794481.614127 MIRROR_RESYNC 4 mirrors 000678 1787794482.800696 write 0x03b336 thru 0x03ffff (0x4cca bytes) 000679 1787794482.860800 mapread 0x035ce4 thru 0x03ffff (0xa31c bytes) 000680 1787794482.921963 read 0x028d76 thru 0x031216 (0x84a1 bytes) 000681 1787794482.942332 trunc from 0x040000 to 0x0185b1 (0x27a4e bytes) 000682 1787794482.985229 mapread 0x00c807 thru 0x0185b1 (0xbdab bytes) 000683 1787794483.013889 read 0x008e27 thru 0x0166c6 (0xd8a0 bytes) 000684 1787794483.016974 write 0x01dea1 thru 0x023081 (0x51e1 bytes) 000685 1787794483.021884 trunc from 0x023082 to 0x034ac9 (0x11a48 bytes) 000687 1787794483.065112 mapwrite 0x03bf63 thru 0x03ffff (0x409d bytes) 000688 1787794483.171686 MIRROR_RESYNC 4 mirrors 000689 1787794484.940142 trunc from 0x040000 to 0x00e006 (0x31ff9 bytes) 000691 1787794485.021236 read 0x002ff3 thru 0x00a630 (0x763e bytes) 000692 1787794485.036265 trunc from 0x00e007 to 0x00f0ae (0x10a8 bytes) 000693 1787794485.075810 write 0x018c3f thru 0x0250a6 (0xc468 bytes) 000695 1787794485.081057 punch from 0x01dc2f to 0x0250a6 (0x7478 bytes) 000696 1787794485.100930 mapwrite 0x0302f8 thru 0x034cfc (0x4a05 bytes) 000697 1787794485.154040 trunc from 0x034cfd to 0x01c0ff (0x18bfd bytes) 000698 1787794485.179868 trunc from 0x01c100 to 0x015cfd (0x6402 bytes) 000699 1787794485.197701 read 0x007eb3 thru 0x00c430 (0x457e bytes) 000700 1787794485.207808 fallocate 0x034347 thru 0x03dc8c (0x9946 bytes) 000701 1787794485.213554 trunc from 0x03dc8d to 0x0299ae (0x142de bytes) 000702 1787794485.230196 read 0x003c8d thru 0x010a7b (0xcdef bytes) 000704 1787794485.233209 fallocate 0x021661 thru 0x0255f0 (0x3f90 bytes) 000705 1787794485.238185 fallocate 0x02cfb5 thru 0x031fce (0x501a bytes) 000706 1787794485.244067 trunc from 0x02654d to 0x031cda (0xb78e bytes) 000707 1787794485.251066 read 0x02b713 thru 0x02c17e (0xa6c bytes) 000708 1787794485.262450 trunc from 0x031fcf to 0x00974a (0x28884 bytes) 000709 1787794485.281788 write 0x007ac0 thru 0x016551 (0xea92 bytes) 000710 1787794485.284432 trunc from 0x007065 to 0x00db2b (0x6ac7 bytes) 000711 1787794485.330560 mapwrite 0x0385e8 thru 0x039a6f (0x1488 bytes) 000712 1787794485.394406 trunc from 0x007a97 to 0x01320f (0xb779 bytes) 000713 1787794485.409986 write 0x02ddd1 thru 0x03c8aa (0xeada bytes) 000714 1787794485.425992 read 0x013b69 thru 0x01f461 (0xb8f9 bytes) 000715 1787794485.458791 fallocate 0x00bc82 thru 0x00f071 (0x33f0 bytes) 000716 1787794485.474585 mapwrite 0x0006e0 thru 0x004154 (0x3a75 bytes) 000717 1787794485.511882 mapwrite 0x01004c thru 0x0141f0 (0x41a5 bytes) 000718 1787794485.577558 read 0x030dd4 thru 0x037b95 (0x6dc2 bytes) 000720 1787794485.580356 punch from 0x011cd1 to 0x01c2fc (0xa62c bytes) 000721 1787794485.595027 mapread 0x00b579 thru 0x0151a4 (0x9c2c bytes) 000723 1787794485.642933 fallocate 0x03d3ff thru 0x03ffff (0x2c01 bytes) 000725 1787794485.650754 mapread 0x025f56 thru 0x0285e6 (0x2691 bytes) 000726 1787794485.652313 trunc from 0x023987 to 0x02e922 (0xaf9c bytes) 000727 1787794485.657953 trunc from 0x03c8ab to 0x00289e (0x3a00c bytes) 000728 1787794485.687018 mapwrite 0x0377fb thru 0x03ffff (0x8805 bytes) 000729 1787794485.832101 mapread 0x0335dc thru 0x03e28b (0xacb0 bytes) 000730 1787794485.878956 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x18586b77 (stale) Mirror 5: 0x18586b77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x18586b77 (stale) Mirror 5: 0x18586b77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000731 1787794486.283560 trunc from 0x040000 to 0x02fb3c (0x104c3 bytes) 000732 1787794486.323169 mapwrite 0x01100f thru 0x0114cb (0x4bd bytes) 000733 1787794486.374810 MIRROR_RESYNC 3 mirrors 000734 1787794487.692761 mapread 0x02f696 thru 0x02fb3c (0x4a7 bytes) 000735 1787794487.707967 trunc from 0x0254f0 to 0x02fb3c (0xa64d bytes) 000736 1787794487.774666 mapread 0x00573d thru 0x0135db (0xde9f bytes) 000737 1787794487.807443 punch from 0x02ac4c to 0x02fb3c (0x4ef1 bytes) 000738 1787794487.814770 fallocate 0x00f485 thru 0x010a70 (0x15ec bytes) 000739 1787794487.821052 trunc from 0x014254 to 0x014560 (0x30d bytes) 000740 1787794487.828839 trunc from 0x02d9ae to 0x02fb3c (0x218f bytes) 000741 1787794487.835768 punch from 0x0133df to 0x016a41 (0x3663 bytes) 000742 1787794487.843761 mapwrite 0x02e188 thru 0x03550c (0x7385 bytes) 000743 1787794487.974459 punch from 0x01d398 to 0x02c267 (0xeed0 bytes) 000744 1787794487.980626 mapwrite 0x0235b9 thru 0x026bf6 (0x363e bytes) 000745 1787794488.029222 mapread 0x00219b thru 0x004815 (0x267b bytes) 000746 1787794488.030857 punch from 0x00fef6 to 0x01dceb (0xddf6 bytes) 000747 1787794488.050919 read 0x011aaa thru 0x016dca (0x5321 bytes) 000748 1787794488.076721 trunc from 0x03047b to 0x03550c (0x5092 bytes) 000749 1787794488.084880 trunc from 0x03550d to 0x02ff5f (0x55ad bytes) 000751 1787794488.135615 mapread 0x02ac16 thru 0x02ff5f (0x534a bytes) 000752 1787794488.150204 trunc from 0x01cef7 to 0x02419b (0x72a5 bytes) 000753 1787794488.159838 write 0x00aba4 thru 0x016503 (0xb960 bytes) 000754 1787794488.169124 read 0x009d59 thru 0x0165f8 (0xc8a0 bytes) 000755 1787794488.171711 trunc from 0x012e5e to 0x016c3c (0x3ddf bytes) 000756 1787794488.200904 read 0x02ebe1 thru 0x02ff5f (0x137f bytes) 000758 1787794488.205585 fallocate 0x0253d1 thru 0x027f70 (0x2ba0 bytes) 000759 1787794488.226882 write 0x03e296 thru 0x03ffff (0x1d6a bytes) 000760 1787794488.229915 write 0x009889 thru 0x009e54 (0x5cc bytes) 000761 1787794488.233613 read 0x019b2f thru 0x01bd49 (0x221b bytes) 000762 1787794488.258767 trunc from 0x0010f7 to 0x001237 (0x141 bytes) 000763 1787794488.280817 mapread 0x02ea29 thru 0x035949 (0x6f21 bytes) 000764 1787794488.282052 write 0x01d790 thru 0x01e53c (0xdad bytes) 000765 1787794488.286033 trunc from 0x000a79 to 0x00cd3f (0xc2c7 bytes) 000766 1787794488.325714 read 0x024348 thru 0x02d072 (0x8d2b bytes) 000767 1787794488.326674 trunc from 0x040000 to 0x034ad2 (0xb52d bytes) 000769 1787794488.404960 MIRROR_RESYNC 3 mirrors 000770 1787794489.965981 mapwrite 0x039878 thru 0x03e306 (0x4a8f bytes) 000771 1787794490.071394 trunc from 0x02d7de to 0x032e48 (0x566b bytes) 000772 1787794490.077454 mapread 0x02b44e thru 0x02c6f6 (0x12a9 bytes) 000773 1787794490.104827 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e307): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e307): Mirror 4: 0xe006617a (stale) Mirror 5: 0xe006617a (stale) CRC-32 checksum value for chunk [0, 0x3e307): Mirror 1: 0x781d4f73 Mirror 6: 0x781d4f73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e307: skipped 000774 1787794491.819125 read 0x01bd29 thru 0x01c783 (0xa5b bytes) 000775 1787794491.826670 fallocate 0x02d044 thru 0x0336b9 (0x6676 bytes) 000776 1787794491.883195 punch from 0x017718 to 0x023a11 (0xc2fa bytes) 000777 1787794491.891921 mapread 0x00aa52 thru 0x0140a0 (0x964f bytes) 000778 1787794491.919829 write 0x02a944 thru 0x02b576 (0xc33 bytes) 000779 1787794491.947886 read 0x006833 thru 0x006d2c (0x4fa bytes) 000780 1787794491.948864 trunc from 0x03e307 to 0x0397b0 (0x4b56 bytes) 000781 1787794492.038402 trunc from 0x0128a2 to 0x01b588 (0x8ce7 bytes) 000782 1787794492.065305 MIRROR_RESYNC 4 mirrors 000783 1787794494.465174 trunc from 0x014cd9 to 0x014e4e (0x176 bytes) 000784 1787794494.523388 write 0x027291 thru 0x0371c6 (0xff36 bytes) 000787 1787794494.539079 mapwrite 0x0061d7 thru 0x00dbeb (0x7a15 bytes) 000788 1787794494.592588 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x397b1): Mirror 4: 0x23bfe23d (stale) Mirror 5: 0x23bfe23d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397b1): Mirror 4: 0x23bfe23d (stale) Mirror 5: 0x23bfe23d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397b1: skipped 000789 1787794495.000738 punch from 0x03395e to 0x0397b0 (0x5e53 bytes) 000790 1787794495.010075 mapread 0x00177e thru 0x0066e4 (0x4f67 bytes) 000791 1787794495.029448 mapread 0x020794 thru 0x02d31f (0xcb8c bytes) 000792 1787794495.059559 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x397b1): Mirror 5: 0x23bfe23d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397b1): Mirror 5: 0x23bfe23d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397b1: skipped 000793 1787794495.437669 mapread 0x023d34 thru 0x0316fb (0xd9c8 bytes) 000794 1787794495.541836 punch from 0x00bc9f to 0x018b0c (0xce6e bytes) 000795 1787794495.589147 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000796 1787794495.900366 read 0x01fea1 thru 0x029558 (0x96b8 bytes) 000797 1787794495.930716 mapread 0x02b70f thru 0x034a93 (0x9385 bytes) 000798 1787794495.970821 fallocate 0x03cce9 thru 0x03f73b (0x2a53 bytes) 000799 1787794495.993769 mapwrite 0x010806 thru 0x01b56c (0xad67 bytes) 000801 1787794496.108180 fallocate 0x0162e8 thru 0x017b2a (0x1843 bytes) 000802 1787794496.115149 read 0x004bee thru 0x0066da (0x1aed bytes) 000804 1787794496.137630 trunc from 0x0397b1 to 0x01d857 (0x1bf59 bytes) 000805 1787794496.247826 mapread 0x0072df thru 0x010f22 (0x9c44 bytes) 000806 1787794496.249990 punch from 0x00f29f to 0x00fad7 (0x839 bytes) 000807 1787794496.260864 trunc from 0x01ba2c to 0x01d857 (0x1e2c bytes) 000808 1787794496.273960 write 0x002fb5 thru 0x00c6d0 (0x971c bytes) 000809 1787794496.277225 read 0x019e12 thru 0x01d857 (0x3a46 bytes) 000810 1787794496.304982 trunc from 0x01d858 to 0x00f472 (0xe3e5 bytes) 000811 1787794496.365344 fallocate 0x01d9d8 thru 0x02a017 (0xc640 bytes) 000812 1787794496.377587 mapread 0x00de95 thru 0x010145 (0x22b1 bytes) 000813 1787794496.407508 fallocate 0x01db52 thru 0x025dd5 (0x8284 bytes) 000816 1787794496.441795 trunc from 0x01cc04 to 0x0296a9 (0xcaa6 bytes) 000817 1787794496.452512 mapread 0x0069d7 thru 0x012eb4 (0xc4de bytes) 000819 1787794496.483160 trunc from 0x02a018 to 0x03551c (0xb505 bytes) 000820 1787794496.543399 mapread 0x00651b thru 0x010cab (0xa791 bytes) 000821 1787794496.544542 mapwrite 0x039721 thru 0x03ffff (0x68df bytes) 000822 1787794496.680201 mapwrite 0x017e0f thru 0x027969 (0xfb5b bytes) 000823 1787794497.004566 read 0x013466 thru 0x0159a9 (0x2544 bytes) 000825 1787794497.023324 mapread 0x032dc3 thru 0x0339b9 (0xbf7 bytes) 000829 1787794497.033463 punch from 0x01278b to 0x01d14b (0xa9c1 bytes) 000830 1787794497.043138 write 0x01a149 thru 0x025e31 (0xbce9 bytes) 000832 1787794497.053961 punch from 0x018e90 to 0x01f93e (0x6aaf bytes) 000834 1787794497.084803 trunc from 0x0029d0 to 0x009955 (0x6f86 bytes) 000835 1787794497.108859 trunc from 0x040000 to 0x034944 (0xb6bb bytes) 000836 1787794497.128745 trunc from 0x034945 to 0x03c97e (0x803a bytes) 000837 1787794497.186916 MIRROR_EXTEND to 2 mirrors 000838 1787794498.730013 punch from 0x0327c8 to 0x03b8f1 (0x912a bytes) 000839 1787794498.793369 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000840 1787794499.004446 read 0x007dce thru 0x013a96 (0xbcc9 bytes) 000841 1787794499.037421 trunc from 0x03c97f to 0x03982e (0x3150 bytes) 000843 1787794499.090648 mapwrite 0x02dcf0 thru 0x035771 (0x7a82 bytes) 000844 1787794499.165223 mapread 0x01bf07 thru 0x0217af (0x58a9 bytes) 000845 1787794499.178693 read 0x00165e thru 0x006852 (0x51f5 bytes) 000846 1787794499.202850 mapwrite 0x02b643 thru 0x0337d2 (0x8190 bytes) 000847 1787794499.224825 mapwrite 0x016fc2 thru 0x01f438 (0x8477 bytes) 000848 1787794499.260701 write 0x03d292 thru 0x03ffff (0x2d6e bytes) 000850 1787794499.270447 write 0x025161 thru 0x031595 (0xc435 bytes) 000851 1787794499.278077 mapwrite 0x035d0c thru 0x03a7f7 (0x4aec bytes) 000852 1787794499.323674 write 0x034147 thru 0x03ffff (0xbeb9 bytes) 000854 1787794499.330133 trunc from 0x040000 to 0x02d354 (0x12cab bytes) 000855 1787794499.391947 write 0x03f3c8 thru 0x03ffff (0xc38 bytes) 000856 1787794499.394072 mapread 0x01e991 thru 0x02d315 (0xe985 bytes) 000857 1787794499.396239 fallocate 0x00aac9 thru 0x014482 (0x99ba bytes) 000860 1787794499.442704 MIRROR_EXTEND to 2 mirrors 000861 1787794500.869007 read 0x031bd8 thru 0x035489 (0x38b2 bytes) 000862 1787794500.881708 read 0x0029c4 thru 0x00c10a (0x9747 bytes) 000863 1787794500.898799 MIRROR_RESYNC 2 mirrors 000864 1787794501.671966 write 0x00baeb thru 0x013897 (0x7dad bytes) 000865 1787794501.765936 punch from 0x0120dc to 0x01bd17 (0x9c3c bytes) 000866 1787794501.785691 mapwrite 0x034253 thru 0x038a5b (0x4809 bytes) 000867 1787794501.831510 mapread 0x00e213 thru 0x017b00 (0x98ee bytes) 000868 1787794501.847332 mapread 0x018a44 thru 0x021fe7 (0x95a4 bytes) 000871 1787794501.869376 trunc from 0x03f098 to 0x03f0f0 (0x59 bytes) 000872 1787794501.887770 write 0x037f63 thru 0x0392e2 (0x1380 bytes) 000873 1787794501.891575 mapread 0x009e61 thru 0x0174dd (0xd67d bytes) 000874 1787794501.917773 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1b9dd445 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb6a7a556 Mirror 3: 0xb6a7a556 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000875 1787794503.729607 read 0x012483 thru 0x013cb9 (0x1837 bytes) 000876 1787794503.744883 mapread 0x030c3c thru 0x03cdfa (0xc1bf bytes) 000877 1787794503.832616 punch from 0x003c55 to 0x011395 (0xd741 bytes) 000878 1787794503.953289 trunc from 0x03084b to 0x03771e (0x6ed4 bytes) 000879 1787794503.981235 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb6a7a556 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb6a7a556 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000880 1787794504.506208 fallocate 0x03128f thru 0x03ffff (0xed71 bytes) 000881 1787794504.526639 mapwrite 0x02f6e0 thru 0x034314 (0x4c35 bytes) 000882 1787794504.624526 mapwrite 0x009e3f thru 0x00da20 (0x3be2 bytes) 000884 1787794504.684622 mapread 0x034b0c thru 0x03ffff (0xb4f4 bytes) 000885 1787794504.741012 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xb6a7a556 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x424f7cca Mirror 4: 0x424f7cca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000886 1787794506.729256 trunc from 0x0096d7 to 0x01471d (0xb047 bytes) 000888 1787794506.808096 read 0x02aec4 thru 0x034cf3 (0x9e30 bytes) 000889 1787794506.825883 mapread 0x035b57 thru 0x03ffff (0xa4a9 bytes) 000890 1787794506.860129 fallocate 0x02cc08 thru 0x02ee60 (0x2259 bytes) 000891 1787794506.871807 mapread 0x02c6cc thru 0x03bc62 (0xf597 bytes) 000892 1787794506.882801 trunc from 0x028963 to 0x02dcbf (0x535d bytes) 000893 1787794506.919278 MIRROR_RESYNC 3 mirrors 000894 1787794509.223953 punch from 0x02dcb0 to 0x03b470 (0xd7c1 bytes) 000896 1787794509.317802 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x162cb15e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x162cb15e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000897 1787794509.808149 read 0x014cd7 thru 0x024474 (0xf79e bytes) 000898 1787794509.824105 fallocate 0x02a1f8 thru 0x02da85 (0x388e bytes) 000899 1787794509.837004 MIRROR_RESYNC 2 mirrors 000900 1787794510.954040 trunc from 0x040000 to 0x01155a (0x2eaa5 bytes) 000901 1787794511.001039 mapwrite 0x0287a3 thru 0x02d41a (0x4c78 bytes) 000902 1787794511.061531 punch from 0x010ca7 to 0x01cc3b (0xbf95 bytes) 000903 1787794511.067428 write 0x019cc0 thru 0x022f18 (0x9259 bytes) 000904 1787794511.079298 punch from 0x00fd90 to 0x01c666 (0xc8d7 bytes) 000905 1787794511.093036 fallocate 0x026621 thru 0x02af5e (0x493e bytes) 000906 1787794511.101237 trunc from 0x02d41b to 0x03f8d0 (0x124b6 bytes) 000908 1787794511.122976 fallocate 0x004a58 thru 0x0105d0 (0xbb79 bytes) 000909 1787794511.129350 mapread 0x032591 thru 0x03347e (0xeee bytes) 000910 1787794511.141357 trunc from 0x03f8d1 to 0x0283ee (0x174e2 bytes) 000912 1787794511.157759 read 0x017f37 thru 0x0201cb (0x8295 bytes) 000913 1787794511.167055 mapwrite 0x01337c thru 0x01f17f (0xbe04 bytes) 000914 1787794511.200350 read 0x006742 thru 0x0141cb (0xda8a bytes) 000915 1787794511.219639 write 0x01286d thru 0x016e7d (0x4611 bytes) 000916 1787794511.222159 write 0x022c5e thru 0x02df94 (0xb337 bytes) 000917 1787794511.224153 read 0x01e4da thru 0x01fa6b (0x1592 bytes) 000918 1787794511.225527 write 0x009b62 thru 0x018019 (0xe4b8 bytes) 000919 1787794511.228311 punch from 0x010b8c to 0x01206f (0x14e4 bytes) 000920 1787794511.269249 trunc from 0x017c9a to 0x0187ab (0xb12 bytes) 000921 1787794511.298392 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2df95): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df95): Mirror 3: 0x2dff29e1 (stale) CRC-32 checksum value for chunk [0, 0x2df95): Mirror 1: 0xdaf4a304 Mirror 4: 0xdaf4a304 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df95: skipped 000922 1787794512.409783 mapread 0x01a8eb thru 0x02a6ed (0xfe03 bytes) 000924 1787794512.478080 write 0x01689f thru 0x02428e (0xd9f0 bytes) 000925 1787794512.580280 fallocate 0x01c419 thru 0x026e3c (0xaa24 bytes) 000926 1787794512.587025 punch from 0x01594c to 0x01b415 (0x5aca bytes) 000927 1787794512.606051 mapwrite 0x03a0be thru 0x03ffff (0x5f42 bytes) 000929 1787794512.721771 mapwrite 0x00fe08 thru 0x016dfd (0x6ff6 bytes) 000930 1787794512.775953 write 0x01183d thru 0x0155a1 (0x3d65 bytes) 000932 1787794512.791241 MIRROR_RESYNC 3 mirrors 000933 1787794514.818996 punch from 0x03f996 to 0x03ffff (0x66a bytes) 000934 1787794514.862369 trunc from 0x03833c to 0x03ffff (0x7cc4 bytes) 000935 1787794514.868847 read 0x02d717 thru 0x03d21f (0xfb09 bytes) 000936 1787794514.883284 trunc from 0x0154ad to 0x01a2c2 (0x4e16 bytes) 000937 1787794514.896833 read 0x00919e thru 0x00b4aa (0x230d bytes) 000938 1787794514.911184 mapread 0x01e04b thru 0x02c7f4 (0xe7aa bytes) 000939 1787794514.913408 read 0x016d70 thru 0x018814 (0x1aa5 bytes) 000940 1787794514.914729 trunc from 0x040000 to 0x016978 (0x29687 bytes) 000942 1787794514.938581 trunc from 0x016979 to 0x00439c (0x125dc bytes) 000943 1787794514.968420 MIRROR_RESYNC 3 mirrors 000944 1787794516.024293 fallocate 0x00c4c2 thru 0x01174d (0x528c bytes) 000945 1787794516.108101 trunc from 0x01174e to 0x03fe86 (0x2e739 bytes) 000946 1787794516.134822 mapwrite 0x03fed8 thru 0x03ffff (0x128 bytes) 000947 1787794516.181030 trunc from 0x040000 to 0x0320e8 (0xdf17 bytes) 000948 1787794516.200861 trunc from 0x011e50 to 0x013036 (0x11e7 bytes) 000949 1787794516.208443 trunc from 0x0320e9 to 0x03a7d2 (0x86ea bytes) 000950 1787794516.229861 read 0x014c5f thru 0x016671 (0x1a13 bytes) 000951 1787794516.237267 mapread 0x02fbb0 thru 0x03a7d2 (0xac23 bytes) 000953 1787794516.269715 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a7d3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7d3): Mirror 3: 0xff30ee0a (stale) Mirror 4: 0xff30ee0a (stale) CRC-32 checksum value for chunk [0, 0x3a7d3): Mirror 1: 0xff30ee0a Mirror 5: 0xff30ee0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7d3: skipped 000954 1787794517.455274 punch from 0x037dff to 0x039f85 (0x2187 bytes) 000955 1787794517.496110 trunc from 0x02ceb1 to 0x02ee9e (0x1fee bytes) 000956 1787794517.506259 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a7d3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7d3): Mirror 3: 0xff30ee0a (stale) Mirror 4: 0xff30ee0a (stale) Mirror 5: 0xff30ee0a (stale) CRC-32 checksum value for chunk [0, 0x3a7d3): Mirror 1: 0xff30ee0a Mirror 6: 0xff30ee0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7d3: skipped 000957 1787794518.787099 write 0x01e339 thru 0x020372 (0x203a bytes) 000959 1787794518.834022 trunc from 0x039c32 to 0x03a7d2 (0xba1 bytes) 000960 1787794518.839553 trunc from 0x009e7d to 0x00a95a (0xade bytes) 000961 1787794518.845164 read 0x022830 thru 0x02ba4b (0x921c bytes) 000962 1787794518.854157 punch from 0x030d70 to 0x03a7d2 (0x9a63 bytes) 000964 1787794518.859779 trunc from 0x0034ad to 0x0096ae (0x6202 bytes) 000965 1787794518.865906 read 0x00a732 thru 0x015b62 (0xb431 bytes) 000966 1787794518.885711 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a7d3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7d3): Mirror 3: 0xff30ee0a (stale) Mirror 4: 0xff30ee0a (stale) Mirror 5: 0xff30ee0a (stale) Mirror 6: 0xff30ee0a (stale) CRC-32 checksum value for chunk [0, 0x3a7d3): Mirror 1: 0xf8930993 Mirror 7: 0xf8930993 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7d3: skipped 000967 1787794520.357836 trunc from 0x03a7d3 to 0x009753 (0x3107f bytes) 000968 1787794520.465344 read 0x007c2e thru 0x009753 (0x1b26 bytes) 000969 1787794520.473398 trunc from 0x0032bd to 0x009753 (0x6497 bytes) 000970 1787794520.484207 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x9754): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9754): Mirror 3: 0x395b691a (stale) Mirror 4: 0x395b691a (stale) Mirror 5: 0x395b691a (stale) Mirror 6: 0x395b691a (stale) Mirror 7: 0x395b691a (stale) CRC-32 checksum value for chunk [0, 0x9754): Mirror 1: 0x395b691a Mirror 8: 0x395b691a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9754: skipped 000971 1787794521.907293 mapread 0x001291 thru 0x009753 (0x84c3 bytes) 000972 1787794521.958547 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9754): Mirror 4: 0x395b691a (stale) Mirror 5: 0x395b691a (stale) Mirror 6: 0x395b691a (stale) Mirror 7: 0x395b691a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9754): Mirror 4: 0x395b691a (stale) Mirror 5: 0x395b691a (stale) Mirror 6: 0x395b691a (stale) Mirror 7: 0x395b691a (stale) CRC-32 checksum value for chunk [0, 0x9754): Mirror 1: 0x395b691a Mirror 8: 0x395b691a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9754: skipped 000973 1787794522.692203 read 0x00153b thru 0x006e83 (0x5949 bytes) 000974 1787794522.710772 MIRROR_RESYNC 6 mirrors 000975 1787794524.468742 MIRROR_EXTEND to 7 mirrors 000976 1787794525.617304 trunc from 0x00176c to 0x009753 (0x7fe8 bytes) 000977 1787794525.706869 MIRROR_RESYNC 7 mirrors 000978 1787794527.020397 fallocate 0x02909c thru 0x035d47 (0xccac bytes) 000979 1787794527.084981 punch from 0x0221f6 to 0x02b8ea (0x96f5 bytes) 000980 1787794527.101414 MIRROR_RESYNC 7 mirrors 000981 1787794528.503996 mapread 0x027a9f thru 0x030eb5 (0x9417 bytes) 000982 1787794528.526458 punch from 0x013340 to 0x0228a4 (0xf565 bytes) 000983 1787794528.580368 mapread 0x01bc2c thru 0x01d404 (0x17d9 bytes) 000985 1787794528.601036 fallocate 0x020935 thru 0x02461b (0x3ce7 bytes) 000986 1787794528.606947 write 0x01decc thru 0x02b4a3 (0xd5d8 bytes) 000987 1787794528.613930 read 0x0194e5 thru 0x01df9b (0x4ab7 bytes) 000988 1787794528.614768 mapwrite 0x0330eb thru 0x0376b1 (0x45c7 bytes) 000989 1787794528.670768 MIRROR_RESYNC 7 mirrors 000990 1787794530.564934 punch from 0x02971d to 0x02d970 (0x4254 bytes) 000991 1787794530.624150 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x376b2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376b2): Mirror 4: 0x68b30403 (stale) Mirror 5: 0x68b30403 (stale) Mirror 6: 0x68b30403 (stale) Mirror 7: 0x68b30403 (stale) Mirror 8: 0x68b30403 (stale) Mirror 9: 0x68b30403 (stale) CRC-32 checksum value for chunk [0, 0x376b2): Mirror 1: 0x85af04fd Mirror 10: 0x85af04fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376b2: skipped 000992 1787794532.031277 punch from 0x0149a1 to 0x01bff0 (0x7650 bytes) 000993 1787794532.092293 read 0x02e937 thru 0x036082 (0x774c bytes) 000994 1787794532.103188 write 0x03874c thru 0x03ffff (0x78b4 bytes) 000997 1787794532.114145 mapwrite 0x006d20 thru 0x012d0d (0xbfee bytes) 000998 1787794532.180945 fallocate 0x02171d thru 0x02462b (0x2f0f bytes) 000999 1787794532.186137 trunc from 0x0022e5 to 0x0077c2 (0x54de bytes) 001000 1787794532.192409 trunc from 0x040000 to 0x038207 (0x7df8 bytes) All operations completed A-OK! PASS 70a (142s) == sanity-flr test 71: check mirror extend parent fid ==== 21:35:38 (1787794538) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x159:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 21:35:49 (1787794549) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.283809 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083216 s, 12.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 21:36:00 (1787794560) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8019488 .. Extending file size to 7889504 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8869792 .. Extending file size to 8406272 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8497952 .. Extending file size to 8535552 .. Extending file size to 8951104 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62588 62589 62590 62593 62595 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) 192.168.204.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 21:37:21 (1787794641) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8478688 .. Extending file size to 1525056 .. done Extending file size to 7900096 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9252512 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9135712 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 64293 64294 64296 64297 64300 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) 192.168.204.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 21:38:41 (1787794721) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1268852 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24872 3582148 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18996 3588024 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43868 7170172 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1268852 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24872 3582148 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18996 3588024 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43868 7170172 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100705 s, 20.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228327 s, 9.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xaa0e6ea8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 21:39:02 (1787794742) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138759 s, 15.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 21:39:09 (1787794749) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xaa:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 21:39:18 (1787794758) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0895595 s, 23.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 21:39:26 (1787794766) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120053 s, 17.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c9:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 21:39:36 (1787794776) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116549 s, 18.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15c:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 21:40:10 (1787794810) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg440-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099179 s, 10.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x160:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 21:41:02 (1787794862) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg440-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0929477 s, 22.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 21:41:50 (1787794910) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg440-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg440-server: Pool lustre.test_204f created Adding targets to pool oleg440-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg440-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122586 s, 17.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x161:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x161:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg440-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg440-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg440-server: Pool lustre.test_204f destroyed PASS 204f (70s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 21:43:00 (1787794980) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 21:43:09 (1787794989) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 21:43:20 (1787795000) oleg440-server: Pool lustre.test_206 created Adding targets to pool oleg440-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg440-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg440-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg440-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg440-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 21:43:49 (1787795029) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656031 s, 16.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625028 s, 16.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x81def46 Mirror 3: 0x81def46 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 21:44:00 (1787795040) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 21:44:04 (1787795044) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 21:44:09 (1787795049) Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0750254 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2060000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2060000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2060000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1290000 Read this (4 bytes): BBBB 192.168.204.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 21:44:48 (1787795088) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0169543 s, 483 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 21:44:55 (1787795095) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566172 s, 18.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc9 lmm_fid: [0x200000405:0xc9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc9 lmm_fid: [0x200000405:0xc9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611337 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 21:44:57.000000000 -0400 Modify: 2026-08-26 21:44:57.000000000 -0400 Change: 2026-08-26 21:44:58.000000000 -0400 Birth: 2026-08-26 21:44:57.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 21:45:04 (1787795104) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494451 s, 21.2 MB/s oleg440-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg440-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg440-server: O/280000401/d9/169: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 21:45:29 (1787795129) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.215857 s, 190 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.037807 s, 108 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 21:45 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xce:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 21:45 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 21:45:37 (1787795137) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 21:45:44 (1787795144) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1973 sec =========== 21:45:53 (1787795153) === sanity-flr: start cleanup 21:45:55 (1787795155) === === sanity-flr: finish cleanup 21:46:05 (1787795165) ===