-----============= acceptance-small: sanity-flr ============----- Fri May 15 06:01:34 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg123-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:01:57 (1778839317) === oleg123-client.virtnet: executing check_config_client /mnt/lustre oleg123-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg123-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90e948d4e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90e948d4e800.idle_timeout=debug disable quota as required oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 06:02:37 (1778839357) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7587] == sanity-flr test 0a: lfs mirror create with -N option == 06:02:39 (1778839359) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:03:00 (1778839380) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:03:04 (1778839384) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:03:07 (1778839387) PASS 0d (32s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:03:39 (1778839419) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:03:43 (1778839423) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 06:03:47 (1778839427) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff90e948d4e800.stats=clear osc.lustre-OST0001-osc-ffff90e948d4e800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:03:56 (1778839436) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (15s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:04:11 (1778839451) PASS 0j (11s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:04:22 (1778839462) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:04:26 (1778839466) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (11s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:04:37 (1778839477) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 06:04:48 (1778839488) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:04:56 (1778839496) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:05:00 (1778839500) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:05:09 (1778839509) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:05:21 (1778839521) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:05:30 (1778839530) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.64351 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226882 s, 4.6 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:05:45 (1778839545) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.225687 s, 37.2 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:05:54 (1778839554) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:06:06 (1778839566) Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e948d4e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e948d4e800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg123-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90e948d4e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e948d4e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg123-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (139s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:08:25 (1778839705) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.648836 s, 14.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e948d4e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e948d4e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:09:07 (1778839747) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e948d4e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e948d4e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90e948d4e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e948d4e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg123-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (73s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:10:21 (1778839821) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.73446 s, 11.1 MB/s -rw-r--r-- 1 root root 52428800 May 15 06:10 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.997s user 0m0.001s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff90e948d4e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90e948d4e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.400s user 0m0.001s sys 0m0.233s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg123-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg123-server oleg123-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff90e948d4e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e948d4e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.171s user 0m0.000s sys 0m0.207s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg123-server: oleg123-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg123-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg123-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90e948d4e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90e948d4e800.ost_server_uuid in FULL state after 0 sec PASS 33b (88s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:11:49 (1778839909) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:11:53 (1778839913) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:11:57 (1778839917) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:12:01 (1778839921) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 06:12:08 (1778839928) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.627892 s, 20.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:12:21 (1778839941) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637116 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271271 s, 38.7 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:12:29 (1778839949) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:12:36 (1778839956) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00089359 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00171968 s, 595 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104688 s, 978 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00271586 s, 377 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 06:12:48 (1778839968) create test files with size 15743627 .. cd77308b5fe0e8e54fab55406a24ff95 - d58f3535512d3c90efbc9772fd296d03 - 47b0f9288e0ee49ebf0ace9f2406b9af - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg123-client.virtnet /mnt/lustre (opts:) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 06:13:10 (1778839990) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:13:25 (1778840005) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:13:32 (1778840012) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170074 s, 6.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693491 s, 15.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 06:13:41 (1778840021) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118007 s, 17.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.549142 s, 7.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.277503 s, 7.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 06:13:59 (1778840039) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 06:14:03 (1778840043) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:14:07 (1778840047) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343744 s, 30.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 06:14:16 (1778840056) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:14:20 (1778840060) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:14:30 (1778840070) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.290635 s, 36.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg123-client.virtnet /mnt/lustre (opts:) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:14:41 (1778840081) llite.lustre-ffff90e945922000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.26092 s, 40.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg123-client.virtnet /mnt/lustre (opts:) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff90e948834800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:14:53 (1778840093) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:15:33 (1778840133) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:16:04 (1778840164) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 06:16:07 (1778840167) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.247138 s, 12.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.184573 s, 17.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x95474282 (nosync,stale) Mirror 2: 0x95474282 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 204f17b6b4651097c14fa81f3859eb2a - mirror 1 checksum: d836762424b5a6ed05737c382e59895e - mirror 2 checksum: 204f17b6b4651097c14fa81f3859eb2a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 15 06:16:09 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri May 15 06:16:09 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 204f17b6b4651097c14fa81f3859eb2a - mirror 1 checksum: d836762424b5a6ed05737c382e59895e - mirror 2 checksum: d836762424b5a6ed05737c382e59895e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 15 06:16:09 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri May 15 06:16:09 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:16:19 (1778840179) Failing mds1 on oleg123-server Stopping /mnt/lustre-mds1 (opts:) on oleg123-server 06:16:25 (1778840185) shut down facet: mds1 facet_host: oleg123-server facet_failover_host: oleg123-server Failover mds1 to oleg123-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg123-server: oleg123-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg123-client: oleg123-server: ssh exited with exit code 1 Started lustre-MDT0000 06:16:47 (1778840207) targets are mounted 06:16:47 (1778840207) facet_failover done oleg123-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239848 s, 43.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (42s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:17:01 (1778840221) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:17:04 (1778840224) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:17:07 (1778840227) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00636905 s, 2.6 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:17:14 (1778840234) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:17:17 (1778840237) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:17:20 (1778840240) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:17:41 (1778840261) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:18:09 (1778840289) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:18:32 (1778840312) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:18:56 (1778840336) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 06:19:03 (1778840343) mirror create pid 53545 mirror split pid 53547 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53545 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53547 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (65s) == sanity-flr test 70a: flr mode fsx test ================ 06:20:08 (1778840408) Seed set to 410 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1778840410.761888 write 0x011178 thru 0x01b760 (0xa5e9 bytes) 000009 1778840410.770826 read 0x006fa5 thru 0x01063f (0x969b bytes) 000010 1778840410.786247 mapwrite 0x0210d8 thru 0x02acaa (0x9bd3 bytes) 000011 1778840410.868235 read 0x0168b7 thru 0x01b0d3 (0x481d bytes) truncating to largest ever: 0x36068 000012 1778840410.869129 trunc from 0x02acab to 0x036067 (0xb3bd bytes) 000013 1778840410.894545 read 0x00ea8a thru 0x012424 (0x399b bytes) 000014 1778840410.896986 read 0x033fc2 thru 0x036067 (0x20a6 bytes) 000015 1778840410.914548 mapread 0x02574c thru 0x031acb (0xc380 bytes) 000017 1778840410.917962 read 0x026be6 thru 0x02765d (0xa78 bytes) 000020 1778840410.924298 read 0x008e1e thru 0x01819b (0xf37e bytes) 000026 1778840410.938063 mapwrite 0x038b3e thru 0x03ffff (0x74c2 bytes) 000028 1778840411.036232 write 0x012e34 thru 0x019a57 (0x6c24 bytes) 000029 1778840411.037666 mapread 0x014b56 thru 0x01e050 (0x94fb bytes) 000031 1778840411.039949 mapread 0x02a7bd thru 0x033ed7 (0x971b bytes) 000032 1778840411.041362 read 0x00803d thru 0x01412b (0xc0ef bytes) 000033 1778840411.042662 write 0x029a75 thru 0x032f20 (0x94ac bytes) 000034 1778840411.044562 read 0x002434 thru 0x0054c2 (0x308f bytes) 000036 1778840411.046068 write 0x02e107 thru 0x036fb7 (0x8eb1 bytes) 000037 1778840411.050981 MIRROR_EXTEND to 2 mirrors 000038 1778840411.680345 mapread 0x01b3dd thru 0x026c90 (0xb8b4 bytes) 000039 1778840411.719094 mapread 0x00371b thru 0x00b2ce (0x7bb4 bytes) 000040 1778840411.739893 MIRROR_RESYNC 2 mirrors 000041 1778840412.168380 mapwrite 0x010cf3 thru 0x01412c (0x343a bytes) 000042 1778840412.325016 mapread 0x004291 thru 0x00edbe (0xab2e bytes) 000043 1778840412.338139 write 0x00a3cf thru 0x00ba95 (0x16c7 bytes) 000044 1778840412.339903 write 0x02cc51 thru 0x03ac9e (0xe04e bytes) 000047 1778840412.356502 read 0x033b6e thru 0x033fdf (0x472 bytes) 000048 1778840412.357018 read 0x0061d6 thru 0x00a90a (0x4735 bytes) 000049 1778840412.363598 write 0x00e3c7 thru 0x01083d (0x2477 bytes) 000050 1778840412.367687 read 0x010d5d thru 0x01e24e (0xd4f2 bytes) 000052 1778840412.377461 mapread 0x016d45 thru 0x021c59 (0xaf15 bytes) 000054 1778840412.381491 write 0x03284f thru 0x03ffff (0xd7b1 bytes) 000055 1778840412.389427 MIRROR_RESYNC 2 mirrors 000059 1778840413.035285 MIRROR_EXTEND to 3 mirrors 000061 1778840413.729960 MIRROR_EXTEND to 4 mirrors 000062 1778840414.590135 read 0x03e492 thru 0x03ffff (0x1b6e bytes) 000063 1778840414.616850 MIRROR_RESYNC 4 mirrors 000066 1778840415.083987 mapwrite 0x01317c thru 0x019edf (0x6d64 bytes) 000067 1778840415.253409 write 0x0303e3 thru 0x039c85 (0x98a3 bytes) 000068 1778840415.276003 trunc from 0x040000 to 0x00fd2b (0x302d4 bytes) 000069 1778840415.302051 mapread 0x00cf5a thru 0x00fd2b (0x2dd2 bytes) 000071 1778840415.313967 mapwrite 0x024c99 thru 0x02d347 (0x86af bytes) 000073 1778840415.392688 mapread 0x0007ab thru 0x000dc7 (0x61d bytes) 000074 1778840415.401035 write 0x01266c thru 0x020f31 (0xe8c6 bytes) 000075 1778840415.414634 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d348): Mirror 2: 0x6c2f1af6 (stale) Mirror 4: 0x6c2f1af6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d348): Mirror 2: 0x6c2f1af6 (stale) Mirror 4: 0x6c2f1af6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d348: skipped 000076 1778840415.809469 read 0x014ac1 thru 0x015fad (0x14ed bytes) 000078 1778840415.817965 mapwrite 0x01d9da thru 0x023a7e (0x60a5 bytes) 000080 1778840415.895795 mapwrite 0x004f28 thru 0x00992d (0x4a06 bytes) 000082 1778840415.940065 read 0x01a88f thru 0x023f0a (0x967c bytes) 000083 1778840415.944889 MIRROR_RESYNC 3 mirrors 000085 1778840416.662109 read 0x0152ca thru 0x01e03e (0x8d75 bytes) 000086 1778840416.675474 trunc from 0x02d348 to 0x029e67 (0x34e0 bytes) 000087 1778840416.748294 read 0x014fea thru 0x019c06 (0x4c1d bytes) 000089 1778840416.760611 write 0x023e03 thru 0x02728e (0x348c bytes) 000090 1778840416.762574 write 0x0059a6 thru 0x013bcf (0xe22a bytes) 000091 1778840416.769922 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29e68): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29e68): Mirror 2: 0xcba009e4 (stale) Mirror 4: 0xcba009e4 (stale) CRC-32 checksum value for chunk [0, 0x29e68): Mirror 1: 0xc3e53e9d Mirror 5: 0xc3e53e9d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29e68: skipped 000092 1778840417.436323 mapread 0x001bec thru 0x0039cc (0x1de1 bytes) 000095 1778840417.489961 write 0x0137e5 thru 0x016233 (0x2a4f bytes) 000096 1778840417.569884 write 0x03af75 thru 0x03ffff (0x508b bytes) 000097 1778840417.571964 mapread 0x01ee1c thru 0x0228af (0x3a94 bytes) 000098 1778840417.592764 read 0x02a9af thru 0x02c397 (0x19e9 bytes) 000099 1778840417.605276 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x91a91f0 (stale) Mirror 4: 0x91a91f0 (stale) Mirror 5: 0x660c23ae (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x40e07048 Mirror 6: 0x40e07048 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000101 1778840418.391431 mapread 0x031654 thru 0x03ce1b (0xb7c8 bytes) 000102 1778840418.424281 mapread 0x03be14 thru 0x03ffff (0x41ec bytes) 000103 1778840418.431813 mapread 0x0034cf thru 0x00efe7 (0xbb19 bytes) 000105 1778840418.457487 read 0x018a7e thru 0x01956a (0xaed bytes) 000106 1778840418.467164 mapwrite 0x02f772 thru 0x038702 (0x8f91 bytes) 000107 1778840418.632293 read 0x03a352 thru 0x03cda4 (0x2a53 bytes) 000108 1778840418.634760 mapread 0x034584 thru 0x03fdf2 (0xb86f bytes) 000109 1778840418.637685 read 0x00700c thru 0x00c788 (0x577d bytes) 000111 1778840418.656006 read 0x007c85 thru 0x00cd41 (0x50bd bytes) 000112 1778840418.657369 mapwrite 0x0324ed thru 0x03e763 (0xc277 bytes) truncating to largest ever: 0x3ce47 000113 1778840418.727562 trunc from 0x040000 to 0x03ce46 (0x31b9 bytes) 000114 1778840418.790714 read 0x02313e thru 0x02d813 (0xa6d6 bytes) 000117 1778840418.793901 read 0x03ab01 thru 0x03c96b (0x1e6b bytes) 000120 1778840418.796879 mapwrite 0x037471 thru 0x038399 (0xf29 bytes) 000124 1778840418.845665 trunc from 0x03ce47 to 0x00c7a2 (0x306a4 bytes) 000125 1778840418.873715 mapwrite 0x00c405 thru 0x019577 (0xd173 bytes) 000127 1778840418.977997 write 0x00a5cb thru 0x00bc40 (0x1676 bytes) 000128 1778840418.980175 mapwrite 0x015225 thru 0x01d8b9 (0x8695 bytes) 000129 1778840419.056540 write 0x02e1e9 thru 0x03d8db (0xf6f3 bytes) 000130 1778840419.066794 MIRROR_RESYNC 5 mirrors 000133 1778840420.028832 trunc from 0x03d8dc to 0x0342d1 (0x960a bytes) 000134 1778840420.159284 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x342d2): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342d2): Mirror 2: 0xd0c265c6 (stale) Mirror 4: 0xd0c265c6 (stale) Mirror 5: 0xd0c265c6 (stale) Mirror 6: 0xd0c265c6 (stale) CRC-32 checksum value for chunk [0, 0x342d2): Mirror 1: 0xd0c265c6 Mirror 7: 0xd0c265c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342d2: skipped 000135 1778840421.466809 trunc from 0x0342d2 to 0x020c56 (0x1367b bytes) 000136 1778840421.551043 mapread 0x01dfcb thru 0x020c56 (0x2c8c bytes) 000137 1778840421.589674 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x20c57): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20c57): Mirror 2: 0xeceffb82 (stale) Mirror 4: 0xeceffb82 (stale) Mirror 5: 0xeceffb82 (stale) Mirror 6: 0xeceffb82 (stale) Mirror 7: 0xeceffb82 (stale) CRC-32 checksum value for chunk [0, 0x20c57): Mirror 1: 0xeceffb82 Mirror 8: 0xeceffb82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20c57: skipped 000139 1778840422.412354 mapread 0x018827 thru 0x01f5f8 (0x6dd2 bytes) 000140 1778840422.432478 trunc from 0x020c57 to 0x007d6e (0x18ee8 bytes) 000144 1778840422.559526 mapread 0x0024cf thru 0x007d6e (0x58a0 bytes) 000146 1778840422.582870 mapwrite 0x0047a3 thru 0x00d77f (0x8fdd bytes) 000149 1778840422.683716 mapwrite 0x01fc7e thru 0x025d32 (0x60b5 bytes) 000150 1778840422.760001 MIRROR_RESYNC 7 mirrors 000152 1778840423.870463 trunc from 0x025d33 to 0x0261b1 (0x47f bytes) 000155 1778840423.927673 read 0x011c37 thru 0x01a19e (0x8568 bytes) 000157 1778840423.939462 write 0x025458 thru 0x02c906 (0x74af bytes) 000158 1778840423.950597 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c907): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c907): Mirror 2: 0x496af66f (stale) Mirror 4: 0x496af66f (stale) Mirror 5: 0x496af66f (stale) Mirror 6: 0x496af66f (stale) Mirror 7: 0x496af66f (stale) Mirror 8: 0x496af66f (stale) CRC-32 checksum value for chunk [0, 0x2c907): Mirror 1: 0xf19a855 Mirror 9: 0xf19a855 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c907: skipped 000159 1778840424.947152 trunc from 0x02c907 to 0x02f306 (0x2a00 bytes) 000160 1778840425.083924 read 0x0063d6 thru 0x00ac9c (0x48c7 bytes) 000161 1778840425.091008 read 0x021824 thru 0x02f306 (0xdae3 bytes) 000162 1778840425.103443 MIRROR_RESYNC 8 mirrors 000163 1778840426.301268 mapwrite 0x01aecb thru 0x02a77d (0xf8b3 bytes) 000165 1778840426.517411 trunc from 0x02f307 to 0x00256c (0x2cd9a bytes) 000166 1778840426.540660 write 0x03c49a thru 0x03ffff (0x3b66 bytes) 000169 1778840426.544659 mapwrite 0x0138e5 thru 0x018469 (0x4b85 bytes) 000170 1778840426.599265 MIRROR_RESYNC 8 mirrors 000171 1778840428.049570 write 0x030944 thru 0x030ee4 (0x5a1 bytes) 000173 1778840428.130327 mapread 0x02af1e thru 0x03672d (0xb810 bytes) 000177 1778840428.161143 mapwrite 0x01bede thru 0x022b4d (0x6c70 bytes) 000179 1778840428.223664 mapwrite 0x02a926 thru 0x031191 (0x686c bytes) 000181 1778840428.319197 MIRROR_RESYNC 8 mirrors 000182 1778840429.955968 MIRROR_EXTEND to 9 mirrors 000183 1778840431.107045 mapread 0x00bbb8 thru 0x0140f3 (0x853c bytes) 000189 1778840431.205277 MIRROR_RESYNC 9 mirrors 000190 1778840431.970877 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) 000191 1778840432.401499 mapread 0x0383bb thru 0x03ffff (0x7c45 bytes) 000193 1778840432.431593 mapwrite 0x0219d7 thru 0x0262b6 (0x48e0 bytes) 000194 1778840432.612225 read 0x028a94 thru 0x02f74b (0x6cb8 bytes) 000195 1778840432.622141 trunc from 0x040000 to 0x028b7e (0x17481 bytes) 000196 1778840432.648343 MIRROR_RESYNC 8 mirrors 000200 1778840433.797881 write 0x026165 thru 0x029c7e (0x3b1a bytes) 000203 1778840433.864522 write 0x013015 thru 0x022fd5 (0xffc1 bytes) 000205 1778840433.884036 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29c7f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c7f): Mirror 2: 0xb64c7f7d (stale) Mirror 4: 0xb64c7f7d (stale) Mirror 5: 0xb64c7f7d (stale) Mirror 7: 0xb64c7f7d (stale) Mirror 8: 0xb64c7f7d (stale) Mirror 9: 0xb64c7f7d (stale) Mirror 10: 0xb64c7f7d (stale) CRC-32 checksum value for chunk [0, 0x29c7f): Mirror 1: 0x9569c8aa Mirror 11: 0x9569c8aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c7f: skipped 000207 1778840434.934502 write 0x039782 thru 0x03ffff (0x687e bytes) 000209 1778840435.016930 trunc from 0x040000 to 0x032ce5 (0xd31a bytes) 000212 1778840435.043554 write 0x0100e3 thru 0x0167f8 (0x6716 bytes) 000213 1778840435.055251 write 0x0007c6 thru 0x0094e8 (0x8d23 bytes) 000216 1778840435.069087 write 0x016edf thru 0x018566 (0x1688 bytes) 000217 1778840435.084559 MIRROR_RESYNC 9 mirrors 000218 1778840436.679157 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) 000219 1778840437.197448 read 0x028efe thru 0x032ce5 (0x9de8 bytes) 000221 1778840437.210561 mapread 0x01275d thru 0x01c117 (0x99bb bytes) 000224 1778840437.267797 trunc from 0x032ce6 to 0x029c31 (0x90b4 bytes) 000225 1778840437.352532 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29c32): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c32): Mirror 2: 0x47e66ae3 (stale) Mirror 5: 0x47e66ae3 (stale) Mirror 7: 0x47e66ae3 (stale) Mirror 8: 0x47e66ae3 (stale) Mirror 9: 0x47e66ae3 (stale) Mirror 10: 0x47e66ae3 (stale) Mirror 11: 0x47e66ae3 (stale) CRC-32 checksum value for chunk [0, 0x29c32): Mirror 1: 0x47e66ae3 Mirror 12: 0x47e66ae3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c32: skipped 000226 1778840438.380598 MIRROR_RESYNC 9 mirrors 000227 1778840439.468255 MIRROR_EXTEND to 10 mirrors 000229 1778840440.141114 write 0x009082 thru 0x015e80 (0xcdff bytes) 000230 1778840440.226548 mapread 0x01f0e8 thru 0x029c31 (0xab4a bytes) 000231 1778840440.251519 write 0x0382bb thru 0x03ffff (0x7d45 bytes) 000233 1778840440.254196 trunc from 0x040000 to 0x02c6bb (0x13944 bytes) 000234 1778840440.272880 mapwrite 0x00be18 thru 0x016fa1 (0xb18a bytes) 000235 1778840440.363755 write 0x03596a thru 0x03ffff (0xa696 bytes) 000236 1778840440.376846 MIRROR_RESYNC 10 mirrors 000239 1778840441.854363 MIRROR_EXTEND to 11 mirrors 000240 1778840442.684845 MIRROR_RESYNC 11 mirrors 000242 1778840443.361162 mapread 0x00de16 thru 0x014694 (0x687f bytes) 000244 1778840443.379921 trunc from 0x040000 to 0x036108 (0x9ef7 bytes) 000245 1778840443.455564 trunc from 0x036109 to 0x0368f2 (0x7ea bytes) 000246 1778840443.480960 read 0x02445b thru 0x02cd21 (0x88c7 bytes) 000248 1778840443.494896 read 0x01a6c8 thru 0x02643b (0xbd74 bytes) 000249 1778840443.515432 mapwrite 0x0362ad thru 0x03d73e (0x7492 bytes) 000251 1778840443.623216 MIRROR_RESYNC 11 mirrors 000254 1778840445.147942 trunc from 0x03d73f to 0x029d54 (0x139ea bytes) 000256 1778840445.226310 mapwrite 0x003026 thru 0x00e602 (0xb5dd bytes) 000257 1778840445.308662 MIRROR_RESYNC 11 mirrors 000258 1778840446.882519 MIRROR_RESYNC 11 mirrors 000259 1778840447.449142 read 0x0159a9 thru 0x01b7f2 (0x5e4a bytes) 000261 1778840447.461716 read 0x024ac1 thru 0x029d54 (0x5294 bytes) 000263 1778840447.476925 mapwrite 0x005fff thru 0x00978f (0x3791 bytes) 000264 1778840447.671372 write 0x021a9e thru 0x030cad (0xf210 bytes) 000265 1778840447.688534 MIRROR_RESYNC 11 mirrors 000266 1778840449.348700 MIRROR_RESYNC 11 mirrors 000267 1778840450.046489 read 0x00cd03 thru 0x011aef (0x4ded bytes) 000268 1778840450.066172 write 0x01f644 thru 0x025b00 (0x64bd bytes) 000270 1778840450.193974 write 0x006623 thru 0x006ce4 (0x6c2 bytes) 000273 1778840450.208855 read 0x02705c thru 0x030cad (0x9c52 bytes) 000274 1778840450.246552 read 0x00170b thru 0x0105dd (0xeed3 bytes) 000275 1778840450.248307 trunc from 0x030cae to 0x02df51 (0x2d5c bytes) 000278 1778840450.291031 read 0x0197fe thru 0x028071 (0xe874 bytes) 000280 1778840450.293173 write 0x023d80 thru 0x02aacf (0x6d50 bytes) 000281 1778840450.295875 trunc from 0x02df52 to 0x01c57b (0x119d6 bytes) 000283 1778840450.318955 mapread 0x008282 thru 0x00e84c (0x65cb bytes) 000284 1778840450.328899 MIRROR_RESYNC 11 mirrors 000289 1778840451.919857 mapread 0x007760 thru 0x00db06 (0x63a7 bytes) 000291 1778840451.945496 trunc from 0x01c57c to 0x000173 (0x1c408 bytes) 000292 1778840452.057198 mapread 0x000035 thru 0x000173 (0x13f bytes) 000293 1778840452.082900 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x174): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x174): Mirror 2: 0x638de184 (stale) Mirror 5: 0x638de184 (stale) Mirror 7: 0x638de184 (stale) Mirror 8: 0x638de184 (stale) Mirror 9: 0x638de184 (stale) Mirror 10: 0x638de184 (stale) Mirror 11: 0x638de184 (stale) Mirror 12: 0x638de184 (stale) Mirror 13: 0x638de184 (stale) Mirror 14: 0x638de184 (stale) CRC-32 checksum value for chunk [0, 0x174): Mirror 1: 0x638de184 Mirror 15: 0x638de184 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x174: skipped 000294 1778840453.311717 read 0x000024 thru 0x000173 (0x150 bytes) 000296 1778840453.340628 read 0x0000dd thru 0x000173 (0x97 bytes) 000297 1778840453.352704 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x174): Mirror 2: 0x638de184 (stale) Mirror 5: 0x638de184 (stale) Mirror 7: 0x638de184 (stale) Mirror 8: 0x638de184 (stale) Mirror 9: 0x638de184 (stale) Mirror 10: 0x638de184 (stale) Mirror 11: 0x638de184 (stale) Mirror 12: 0x638de184 (stale) Mirror 13: 0x638de184 (stale) Mirror 14: 0x638de184 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x174): Mirror 2: 0x638de184 (stale) Mirror 5: 0x638de184 (stale) Mirror 7: 0x638de184 (stale) Mirror 8: 0x638de184 (stale) Mirror 9: 0x638de184 (stale) Mirror 10: 0x638de184 (stale) Mirror 11: 0x638de184 (stale) Mirror 12: 0x638de184 (stale) Mirror 13: 0x638de184 (stale) Mirror 14: 0x638de184 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x174: skipped 000298 1778840453.896245 mapread 0x000096 thru 0x000173 (0xde bytes) 000299 1778840453.904135 trunc from 0x000174 to 0x027c41 (0x27ace bytes) 000302 1778840453.946465 mapwrite 0x02972f thru 0x038243 (0xeb15 bytes) 000305 1778840454.079905 trunc from 0x038244 to 0x01152a (0x26d19 bytes) 000306 1778840454.096333 read 0x005038 thru 0x00a7d3 (0x579c bytes) 000308 1778840454.117017 read 0x00ae67 thru 0x00d07d (0x2217 bytes) 000311 1778840454.138655 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1152b): Mirror 2: 0x87a08571 (stale) Mirror 5: 0x87a08571 (stale) Mirror 7: 0x87a08571 (stale) Mirror 8: 0x87a08571 (stale) Mirror 9: 0x87a08571 (stale) Mirror 10: 0x87a08571 (stale) Mirror 11: 0x87a08571 (stale) Mirror 12: 0x87a08571 (stale) Mirror 14: 0x87a08571 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1152b): Mirror 2: 0x87a08571 (stale) Mirror 5: 0x87a08571 (stale) Mirror 7: 0x87a08571 (stale) Mirror 8: 0x87a08571 (stale) Mirror 9: 0x87a08571 (stale) Mirror 10: 0x87a08571 (stale) Mirror 11: 0x87a08571 (stale) Mirror 12: 0x87a08571 (stale) Mirror 14: 0x87a08571 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1152b: skipped 000312 1778840454.623280 write 0x00a938 thru 0x00f64e (0x4d17 bytes) 000313 1778840454.633896 trunc from 0x01152b to 0x0185cc (0x70a2 bytes) 000314 1778840454.650887 mapwrite 0x007a01 thru 0x00c25d (0x485d bytes) 000315 1778840454.706596 mapread 0x00c647 thru 0x016cfe (0xa6b8 bytes) 000317 1778840454.728674 write 0x02555d thru 0x032f69 (0xda0d bytes) 000319 1778840454.732631 read 0x018e60 thru 0x02178c (0x892d bytes) 000321 1778840454.746439 trunc from 0x032f6a to 0x02dcf0 (0x5279 bytes) 000322 1778840454.797199 trunc from 0x02dcf1 to 0x01f747 (0xe5a9 bytes) 000325 1778840454.825945 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f748): Mirror 2: 0x7c8e69d3 (stale) Mirror 5: 0x7c8e69d3 (stale) Mirror 7: 0x7c8e69d3 (stale) Mirror 8: 0x7c8e69d3 (stale) Mirror 9: 0x7c8e69d3 (stale) Mirror 11: 0x7c8e69d3 (stale) Mirror 12: 0x7c8e69d3 (stale) Mirror 14: 0x7c8e69d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f748): Mirror 2: 0x7c8e69d3 (stale) Mirror 5: 0x7c8e69d3 (stale) Mirror 7: 0x7c8e69d3 (stale) Mirror 8: 0x7c8e69d3 (stale) Mirror 9: 0x7c8e69d3 (stale) Mirror 11: 0x7c8e69d3 (stale) Mirror 12: 0x7c8e69d3 (stale) Mirror 14: 0x7c8e69d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f748: skipped 000327 1778840455.442806 read 0x004aa6 thru 0x012067 (0xd5c2 bytes) 000328 1778840455.463260 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f748): Mirror 2: 0x7c8e69d3 (stale) Mirror 5: 0x7c8e69d3 (stale) Mirror 7: 0x7c8e69d3 (stale) Mirror 8: 0x7c8e69d3 (stale) Mirror 9: 0x7c8e69d3 (stale) Mirror 12: 0x7c8e69d3 (stale) Mirror 14: 0x7c8e69d3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f748): Mirror 2: 0x7c8e69d3 (stale) Mirror 5: 0x7c8e69d3 (stale) Mirror 7: 0x7c8e69d3 (stale) Mirror 8: 0x7c8e69d3 (stale) Mirror 9: 0x7c8e69d3 (stale) Mirror 12: 0x7c8e69d3 (stale) Mirror 14: 0x7c8e69d3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f748: skipped 000329 1778840456.027416 write 0x002e45 thru 0x008266 (0x5422 bytes) 000331 1778840456.039824 write 0x00a15a thru 0x0165bb (0xc462 bytes) 000333 1778840456.063690 trunc from 0x01f748 to 0x012bbd (0xcb8a bytes) 000334 1778840456.092460 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x12bbe): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12bbe): Mirror 2: 0x43b69a59 (stale) Mirror 5: 0x43b69a59 (stale) Mirror 7: 0x43b69a59 (stale) Mirror 8: 0x43b69a59 (stale) Mirror 9: 0x43b69a59 (stale) Mirror 12: 0x43b69a59 (stale) Mirror 14: 0x43b69a59 (stale) CRC-32 checksum value for chunk [0, 0x12bbe): Mirror 1: 0x29c4a1a8 Mirror 15: 0x29c4a1a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12bbe: skipped 000336 1778840457.208892 write 0x00d3e0 thru 0x01aaac (0xd6cd bytes) 000337 1778840457.289977 read 0x016a7e thru 0x01aaac (0x402f bytes) 000339 1778840457.297284 MIRROR_RESYNC 9 mirrors 000340 1778840458.810651 trunc from 0x01aaad to 0x0070e3 (0x139c9 bytes) 000341 1778840458.918610 trunc from 0x0070e4 to 0x007c1b (0xb38 bytes) 000342 1778840458.943966 write 0x013db7 thru 0x020144 (0xc38e bytes) 000343 1778840458.950471 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20145): Mirror 5: 0x68bb04f (stale) Mirror 7: 0x68bb04f (stale) Mirror 8: 0x68bb04f (stale) Mirror 9: 0x68bb04f (stale) Mirror 12: 0x68bb04f (stale) Mirror 14: 0x68bb04f (stale) Mirror 15: 0x68bb04f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20145): Mirror 5: 0x68bb04f (stale) Mirror 7: 0x68bb04f (stale) Mirror 8: 0x68bb04f (stale) Mirror 9: 0x68bb04f (stale) Mirror 12: 0x68bb04f (stale) Mirror 14: 0x68bb04f (stale) Mirror 15: 0x68bb04f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20145: skipped 000345 1778840459.478028 mapread 0x004a5e thru 0x0109b2 (0xbf55 bytes) 000346 1778840459.505028 mapread 0x01799c thru 0x020144 (0x87a9 bytes) 000347 1778840459.527517 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20145): Mirror 7: 0x68bb04f (stale) Mirror 8: 0x68bb04f (stale) Mirror 9: 0x68bb04f (stale) Mirror 12: 0x68bb04f (stale) Mirror 14: 0x68bb04f (stale) Mirror 15: 0x68bb04f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20145): Mirror 7: 0x68bb04f (stale) Mirror 8: 0x68bb04f (stale) Mirror 9: 0x68bb04f (stale) Mirror 12: 0x68bb04f (stale) Mirror 14: 0x68bb04f (stale) Mirror 15: 0x68bb04f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20145: skipped 000348 1778840460.038503 mapread 0x00acbb thru 0x018de7 (0xe12d bytes) 000349 1778840460.074544 read 0x00f23a thru 0x011e2a (0x2bf1 bytes) 000350 1778840460.075271 read 0x00cd1e thru 0x00dcc7 (0xfaa bytes) 000352 1778840460.079712 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20145): Mirror 8: 0x68bb04f (stale) Mirror 9: 0x68bb04f (stale) Mirror 12: 0x68bb04f (stale) Mirror 14: 0x68bb04f (stale) Mirror 15: 0x68bb04f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20145): Mirror 8: 0x68bb04f (stale) Mirror 9: 0x68bb04f (stale) Mirror 12: 0x68bb04f (stale) Mirror 14: 0x68bb04f (stale) Mirror 15: 0x68bb04f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20145: skipped 000353 1778840460.529817 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20145): Mirror 8: 0x68bb04f (stale) Mirror 9: 0x68bb04f (stale) Mirror 12: 0x68bb04f (stale) Mirror 15: 0x68bb04f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20145): Mirror 8: 0x68bb04f (stale) Mirror 9: 0x68bb04f (stale) Mirror 12: 0x68bb04f (stale) Mirror 15: 0x68bb04f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20145: skipped 000355 1778840460.904544 write 0x00edb2 thru 0x01ea79 (0xfcc8 bytes) 000357 1778840460.941460 MIRROR_RESYNC 5 mirrors 000358 1778840462.114422 read 0x01d1c1 thru 0x020144 (0x2f84 bytes) 000359 1778840462.125628 mapwrite 0x00cf75 thru 0x01b9a5 (0xea31 bytes) 000360 1778840462.351117 mapread 0x002dae thru 0x006ee0 (0x4133 bytes) 000363 1778840462.373699 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20145): Mirror 8: 0x3f7292c8 (stale) Mirror 12: 0x3f7292c8 (stale) Mirror 15: 0x3f7292c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20145): Mirror 8: 0x3f7292c8 (stale) Mirror 12: 0x3f7292c8 (stale) Mirror 15: 0x3f7292c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20145: skipped 000366 1778840462.824969 mapwrite 0x00501d thru 0x00a867 (0x584b bytes) 000367 1778840462.919783 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20145): Mirror 8: 0x3f7292c8 (stale) Mirror 15: 0x3f7292c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20145): Mirror 8: 0x3f7292c8 (stale) Mirror 15: 0x3f7292c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20145: skipped 000369 1778840463.220137 trunc from 0x020145 to 0x00a440 (0x15d04 bytes) 000370 1778840463.245299 read 0x00671e thru 0x00a440 (0x3d23 bytes) truncating to largest ever: 0x3e406 000371 1778840463.256683 trunc from 0x00a441 to 0x03e405 (0x33fc5 bytes) 000372 1778840463.280488 mapread 0x00597a thru 0x014063 (0xe6ea bytes) 000373 1778840463.304556 mapwrite 0x03b536 thru 0x03ffff (0x4aca bytes) 000374 1778840463.363788 mapread 0x0343b9 thru 0x03ab12 (0x675a bytes) 000375 1778840463.364826 read 0x0077da thru 0x010733 (0x8f5a bytes) 000376 1778840463.365348 write 0x03f71f thru 0x03ffff (0x8e1 bytes) 000377 1778840463.366129 mapwrite 0x021046 thru 0x02b590 (0xa54b bytes) 000381 1778840463.431139 write 0x01d49b thru 0x02a929 (0xd48f bytes) 000382 1778840463.434791 mapread 0x0345b3 thru 0x03ffff (0xba4d bytes) 000383 1778840463.437279 trunc from 0x040000 to 0x01ab1f (0x254e0 bytes) 000385 1778840463.477627 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ab20): Mirror 15: 0x391b848 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ab20): Mirror 15: 0x391b848 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ab20: skipped 000386 1778840463.745301 mapwrite 0x015bfc thru 0x020647 (0xaa4c bytes) 000387 1778840463.837606 trunc from 0x020648 to 0x034e5d (0x14816 bytes) 000388 1778840463.859783 mapread 0x01b96c thru 0x029a28 (0xe0bd bytes) 000390 1778840463.874024 trunc from 0x034e5e to 0x02ed2b (0x6132 bytes) 000391 1778840463.891425 mapread 0x00c93a thru 0x019bb9 (0xd280 bytes) 000394 1778840463.908230 mapwrite 0x03f97a thru 0x03ffff (0x686 bytes) 000395 1778840463.975325 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) 000397 1778840464.102122 read 0x001934 thru 0x00ee27 (0xd4f4 bytes) 000401 1778840464.113614 write 0x01c9e7 thru 0x026a03 (0xa01d bytes) 000402 1778840464.126060 write 0x028fb4 thru 0x036465 (0xd4b2 bytes) 000407 1778840464.144235 mapwrite 0x034ed7 thru 0x03ffff (0xb129 bytes) 000408 1778840464.218864 write 0x021eb6 thru 0x02e447 (0xc592 bytes) 000411 1778840464.224059 MIRROR_EXTEND to 2 mirrors 000416 1778840464.902342 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) 000417 1778840465.035700 read 0x005c79 thru 0x0149c1 (0xed49 bytes) 000418 1778840465.045490 read 0x01d22c thru 0x022151 (0x4f26 bytes) 000419 1778840465.083225 read 0x00b655 thru 0x01092a (0x52d6 bytes) 000421 1778840465.085066 mapwrite 0x014951 thru 0x020df3 (0xc4a3 bytes) 000424 1778840465.192139 mapwrite 0x0100d8 thru 0x01e345 (0xe26e bytes) 000426 1778840465.233526 mapread 0x03704c thru 0x03ffff (0x8fb4 bytes) 000432 1778840465.237514 mapwrite 0x03af15 thru 0x03cf13 (0x1fff bytes) 000433 1778840465.268187 mapwrite 0x037a87 thru 0x0397c3 (0x1d3d bytes) 000434 1778840465.342363 write 0x018bc8 thru 0x01a8ed (0x1d26 bytes) 000436 1778840465.345397 mapread 0x00057b thru 0x000e32 (0x8b8 bytes) 000437 1778840465.346638 trunc from 0x040000 to 0x0101e1 (0x2fe1e bytes) 000438 1778840465.410907 trunc from 0x0101e2 to 0x01c1ec (0xc00b bytes) 000439 1778840465.438037 trunc from 0x01c1ed to 0x035239 (0x1904d bytes) 000440 1778840465.462042 write 0x00244a thru 0x00f9f6 (0xd5ad bytes) 000441 1778840465.467753 mapwrite 0x013766 thru 0x021971 (0xe20c bytes) 000443 1778840465.615386 write 0x030ee5 thru 0x03ffff (0xf11b bytes) 000446 1778840465.625302 read 0x00c1b8 thru 0x00fbe2 (0x3a2b bytes) 000447 1778840465.626285 trunc from 0x040000 to 0x029c6e (0x16391 bytes) 000448 1778840465.650537 mapwrite 0x035505 thru 0x03706d (0x1b69 bytes) 000450 1778840465.732823 mapread 0x0004cc thru 0x004422 (0x3f57 bytes) 000451 1778840465.733834 trunc from 0x03706e to 0x0360c5 (0xfa8 bytes) 000454 1778840465.756043 write 0x00c6e8 thru 0x00f681 (0x2f9a bytes) 000463 1778840465.761880 mapread 0x02df3d thru 0x02f27a (0x133e bytes) 000466 1778840465.765003 mapwrite 0x02fdfe thru 0x030897 (0xa9a bytes) 000471 1778840465.814255 write 0x03b88f thru 0x03ffff (0x4771 bytes) 000472 1778840465.821527 MIRROR_EXTEND to 2 mirrors 000473 1778840466.465589 read 0x037a29 thru 0x03ffff (0x85d7 bytes) 000474 1778840466.476509 mapwrite 0x03136f thru 0x03c239 (0xaecb bytes) 000477 1778840466.625765 mapwrite 0x005e7c thru 0x00831e (0x24a3 bytes) 000478 1778840466.679054 trunc from 0x040000 to 0x02bc69 (0x14396 bytes) 000479 1778840466.711205 read 0x01db2a thru 0x02bc69 (0xe140 bytes) 000480 1778840466.723298 write 0x032232 thru 0x03538c (0x315b bytes) 000481 1778840466.724673 trunc from 0x03538d to 0x035e84 (0xaf8 bytes) 000485 1778840466.748024 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35e85): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e85): Mirror 2: 0x76f34b29 (stale) CRC-32 checksum value for chunk [0, 0x35e85): Mirror 1: 0xffdc43da Mirror 3: 0xffdc43da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e85: skipped 000486 1778840467.537320 mapwrite 0x012beb thru 0x014365 (0x177b bytes) 000487 1778840467.774906 mapwrite 0x009862 thru 0x016dd7 (0xd576 bytes) 000488 1778840467.855687 mapwrite 0x03dc5b thru 0x03ffff (0x23a5 bytes) 000492 1778840467.972141 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9e8f153b (stale) Mirror 3: 0x536ddebc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x19006b14 Mirror 4: 0x19006b14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000495 1778840469.224544 read 0x00d322 thru 0x013844 (0x6523 bytes) 000496 1778840469.245193 trunc from 0x040000 to 0x020050 (0x1ffaf bytes) 000497 1778840469.340535 read 0x00d981 thru 0x01ae7f (0xd4ff bytes) 000498 1778840469.349634 mapread 0x01e611 thru 0x020050 (0x1a40 bytes) 000499 1778840469.350923 trunc from 0x020051 to 0x002773 (0x1d8dd bytes) 000500 1778840469.376688 read 0x001014 thru 0x002773 (0x1760 bytes) 000504 1778840469.380144 trunc from 0x002774 to 0x027eba (0x25747 bytes) 000505 1778840469.397495 trunc from 0x027ebb to 0x018f86 (0xef34 bytes) 000507 1778840469.420278 read 0x004829 thru 0x00fb21 (0xb2f9 bytes) 000508 1778840469.431582 mapread 0x014db1 thru 0x018f86 (0x41d6 bytes) 000509 1778840469.434145 mapwrite 0x01edea thru 0x01fc66 (0xe7d bytes) 000510 1778840469.515263 MIRROR_RESYNC 4 mirrors 000512 1778840470.496577 read 0x003e00 thru 0x00f12a (0xb32b bytes) 000514 1778840470.506812 mapread 0x0019cf thru 0x00aab2 (0x90e4 bytes) 000516 1778840470.517838 read 0x004fea thru 0x00566e (0x685 bytes) 000517 1778840470.518455 mapread 0x01d1a4 thru 0x01fc66 (0x2ac3 bytes) 000519 1778840470.530231 write 0x004f89 thru 0x0094c1 (0x4539 bytes) 000520 1778840470.619883 mapwrite 0x03e719 thru 0x03fb64 (0x144c bytes) 000521 1778840470.723580 mapwrite 0x020fd6 thru 0x02acc8 (0x9cf3 bytes) 000523 1778840470.794117 mapwrite 0x0232f3 thru 0x02ea7c (0xb78a bytes) 000524 1778840470.859902 mapread 0x033da3 thru 0x03536e (0x15cc bytes) 000526 1778840470.861436 mapwrite 0x007e2c thru 0x00aba3 (0x2d78 bytes) 000527 1778840470.913782 read 0x024176 thru 0x02638e (0x2219 bytes) 000528 1778840470.914815 read 0x03b68d thru 0x03fb64 (0x44d8 bytes) 000529 1778840470.916449 read 0x0004c0 thru 0x005a7b (0x55bc bytes) 000530 1778840470.934269 trunc from 0x03fb65 to 0x0025c1 (0x3d5a3 bytes) 000531 1778840470.991511 MIRROR_RESYNC 4 mirrors 000532 1778840471.863305 MIRROR_EXTEND to 5 mirrors 000533 1778840472.399951 mapwrite 0x01a8f0 thru 0x02409f (0x97b0 bytes) 000534 1778840472.628835 read 0x01540c thru 0x01f80f (0xa404 bytes) 000535 1778840472.629690 trunc from 0x0240a0 to 0x03c70c (0x1866d bytes) 000537 1778840472.666524 trunc from 0x03c70d to 0x02cfb6 (0xf756 bytes) 000538 1778840472.686883 mapwrite 0x03e4fa thru 0x03ffff (0x1b06 bytes) 000539 1778840472.750909 mapread 0x028cf1 thru 0x02b67c (0x298c bytes) 000540 1778840472.764002 read 0x012d16 thru 0x022493 (0xf77e bytes) 000541 1778840472.777327 trunc from 0x040000 to 0x0325e0 (0xda1f bytes) 000543 1778840472.795365 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x325e1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x325e1): Mirror 2: 0xab3a5f4c (stale) Mirror 3: 0xab3a5f4c (stale) Mirror 4: 0xab3a5f4c (stale) Mirror 5: 0xab3a5f4c (stale) CRC-32 checksum value for chunk [0, 0x325e1): Mirror 1: 0x858792a8 Mirror 6: 0x858792a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x325e1: skipped 000544 1778840473.764955 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x325e1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x325e1): Mirror 2: 0xab3a5f4c (stale) Mirror 3: 0xab3a5f4c (stale) Mirror 4: 0xab3a5f4c (stale) Mirror 5: 0xab3a5f4c (stale) CRC-32 checksum value for chunk [0, 0x325e1): Mirror 1: 0x858792a8 Mirror 6: 0x858792a8 Mirror 7: 0x858792a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x325e1: skipped 000545 1778840474.667255 mapwrite 0x01dd0b thru 0x02dbe1 (0xfed7 bytes) 000554 1778840474.899711 mapread 0x006b59 thru 0x01682b (0xfcd3 bytes) 000555 1778840474.922505 trunc from 0x0325e1 to 0x030a43 (0x1b9d bytes) 000557 1778840474.971125 write 0x01ca52 thru 0x0220a8 (0x5657 bytes) 000559 1778840474.978968 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30a44): Mirror 2: 0xd27f7144 (stale) Mirror 3: 0xd27f7144 (stale) Mirror 4: 0xd27f7144 (stale) Mirror 5: 0xd27f7144 (stale) Mirror 6: 0xcc9d013 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30a44): Mirror 2: 0xd27f7144 (stale) Mirror 3: 0xd27f7144 (stale) Mirror 4: 0xd27f7144 (stale) Mirror 5: 0xd27f7144 (stale) Mirror 6: 0xcc9d013 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30a44: skipped 000561 1778840475.456457 MIRROR_RESYNC 6 mirrors 000562 1778840476.663444 mapwrite 0x0032ef thru 0x004854 (0x1566 bytes) 000564 1778840476.876297 write 0x00e9cd thru 0x01a221 (0xb855 bytes) 000565 1778840476.884200 mapwrite 0x00529b thru 0x008acf (0x3835 bytes) 000566 1778840476.937106 mapwrite 0x0011ac thru 0x001370 (0x1c5 bytes) 000568 1778840476.981460 MIRROR_RESYNC 6 mirrors 000569 1778840477.993713 read 0x0228a8 thru 0x02cf21 (0xa67a bytes) 000570 1778840478.009030 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) 000572 1778840478.359880 trunc from 0x030a44 to 0x033b65 (0x3122 bytes) 000574 1778840478.449483 write 0x024e0d thru 0x0339b1 (0xeba5 bytes) 000575 1778840478.461954 trunc from 0x033b66 to 0x00bd03 (0x27e62 bytes) 000576 1778840478.502992 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xbd04): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbd04): Mirror 3: 0x15dbe5f1 (stale) Mirror 4: 0x15dbe5f1 (stale) Mirror 5: 0x15dbe5f1 (stale) Mirror 6: 0x15dbe5f1 (stale) CRC-32 checksum value for chunk [0, 0xbd04): Mirror 1: 0x15dbe5f1 Mirror 7: 0x15dbe5f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbd04: skipped 000579 1778840479.494907 write 0x01caa5 thru 0x0227d1 (0x5d2d bytes) 000581 1778840479.558506 trunc from 0x0227d2 to 0x01198b (0x10e46 bytes) 000582 1778840479.579044 read 0x00c718 thru 0x01198b (0x5274 bytes) 000583 1778840479.588678 read 0x000e64 thru 0x00d8c2 (0xca5f bytes) 000584 1778840479.596410 read 0x009a64 thru 0x010e95 (0x7432 bytes) 000585 1778840479.597128 write 0x01e920 thru 0x028567 (0x9c48 bytes) 000586 1778840479.598968 trunc from 0x028568 to 0x0366bc (0xe155 bytes) 000588 1778840479.622532 read 0x0195c7 thru 0x01d666 (0x40a0 bytes) 000589 1778840479.635876 write 0x01535e thru 0x024fd8 (0xfc7b bytes) 000590 1778840479.637426 mapwrite 0x00e66d thru 0x018915 (0xa2a9 bytes) 000591 1778840479.733264 read 0x034012 thru 0x0366bc (0x26ab bytes) 000592 1778840479.734585 write 0x0383bb thru 0x03ffff (0x7c45 bytes) 000593 1778840479.737150 mapwrite 0x015ec7 thru 0x01a413 (0x454d bytes) 000594 1778840479.800360 mapread 0x0158a5 thru 0x021867 (0xbfc3 bytes) 000596 1778840479.804852 write 0x0119b7 thru 0x01d658 (0xbca2 bytes) 000598 1778840479.808117 trunc from 0x040000 to 0x03323b (0xcdc4 bytes) 000602 1778840479.837494 trunc from 0x03323c to 0x00b0fe (0x2813d bytes) 000603 1778840479.866755 read 0x002299 thru 0x0025f5 (0x35d bytes) 000606 1778840479.872212 mapwrite 0x01a0fc thru 0x02429c (0xa1a1 bytes) 000608 1778840479.966102 mapread 0x012b76 thru 0x01f84c (0xccd7 bytes) 000609 1778840479.977602 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2429d): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2429d): Mirror 3: 0xe24dbdb8 (stale) Mirror 4: 0xe24dbdb8 (stale) Mirror 5: 0xe24dbdb8 (stale) Mirror 6: 0xe24dbdb8 (stale) Mirror 7: 0x5b0f0f66 (stale) CRC-32 checksum value for chunk [0, 0x2429d): Mirror 1: 0x8db40e91 Mirror 8: 0x8db40e91 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2429d: skipped 000612 1778840480.883493 write 0x017d96 thru 0x01e6c7 (0x6932 bytes) 000613 1778840480.975238 read 0x01924d thru 0x01a879 (0x162d bytes) 000614 1778840480.976007 write 0x00ca28 thru 0x014789 (0x7d62 bytes) 000615 1778840480.993150 read 0x00e133 thru 0x01855c (0xa42a bytes) 000618 1778840481.001992 read 0x01fbe2 thru 0x02377e (0x3b9d bytes) 000619 1778840481.002599 trunc from 0x02429d to 0x02d18e (0x8ef2 bytes) 000621 1778840481.020557 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d18f): Mirror 4: 0x6dd73900 (stale) Mirror 5: 0x6dd73900 (stale) Mirror 6: 0x6dd73900 (stale) Mirror 7: 0x9b06c359 (stale) Mirror 8: 0x69537064 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d18f): Mirror 4: 0x6dd73900 (stale) Mirror 5: 0x6dd73900 (stale) Mirror 6: 0x6dd73900 (stale) Mirror 7: 0x9b06c359 (stale) Mirror 8: 0x69537064 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d18f: skipped 000624 1778840481.487402 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2d18f): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d18f): Mirror 4: 0x6dd73900 (stale) Mirror 5: 0x6dd73900 (stale) Mirror 6: 0x6dd73900 (stale) Mirror 7: 0x9b06c359 (stale) Mirror 8: 0x69537064 (stale) CRC-32 checksum value for chunk [0, 0x2d18f): Mirror 1: 0xe136d4ca Mirror 9: 0xe136d4ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d18f: skipped 000625 1778840482.461541 write 0x03a630 thru 0x03ffff (0x59d0 bytes) 000626 1778840482.643638 mapread 0x0038d9 thru 0x01136f (0xda97 bytes) 000627 1778840482.675909 read 0x02b43c thru 0x031f33 (0x6af8 bytes) 000629 1778840482.686825 mapwrite 0x02f747 thru 0x030386 (0xc40 bytes) 000630 1778840482.764134 mapwrite 0x037d6e thru 0x03ffff (0x8292 bytes) 000633 1778840482.847304 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4cc32928 (stale) Mirror 5: 0x4cc32928 (stale) Mirror 6: 0x4cc32928 (stale) Mirror 7: 0xb6253376 (stale) Mirror 8: 0x6c559057 (stale) Mirror 9: 0xf7cae52a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5006c36a Mirror 10: 0x5006c36a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000637 1778840483.912534 read 0x03374c thru 0x03ffea (0xc89f bytes) 000640 1778840483.951391 MIRROR_RESYNC 8 mirrors 000642 1778840485.452706 read 0x0173b7 thru 0x02636c (0xefb6 bytes) 000643 1778840485.463061 write 0x01b5b1 thru 0x023773 (0x81c3 bytes) 000644 1778840485.558670 write 0x0263d1 thru 0x033f98 (0xdbc8 bytes) 000645 1778840485.587361 mapwrite 0x03c74d thru 0x03dcaa (0x155e bytes) 000646 1778840485.678175 trunc from 0x040000 to 0x004373 (0x3bc8c bytes) 000649 1778840485.700991 mapread 0x00302a thru 0x004373 (0x134a bytes) 000650 1778840485.709230 read 0x003106 thru 0x004373 (0x126e bytes) 000651 1778840485.710045 read 0x002798 thru 0x004373 (0x1bdc bytes) 000654 1778840485.714205 mapread 0x001e70 thru 0x004373 (0x2504 bytes) 000659 1778840485.721049 mapread 0x00282e thru 0x004159 (0x192c bytes) 000660 1778840485.722419 mapread 0x001f52 thru 0x004373 (0x2422 bytes) 000662 1778840485.725542 read 0x000258 thru 0x004373 (0x411c bytes) 000663 1778840485.732136 mapwrite 0x026f36 thru 0x02d813 (0x68de bytes) 000664 1778840485.805690 trunc from 0x02d814 to 0x01d5fd (0x10216 bytes) 000666 1778840485.823845 mapwrite 0x000b7d thru 0x00ae26 (0xa2aa bytes) 000668 1778840485.864408 mapwrite 0x0019f8 thru 0x002aff (0x1108 bytes) 000669 1778840485.902656 write 0x03f256 thru 0x03ffff (0xdaa bytes) 000670 1778840485.904179 mapwrite 0x039c20 thru 0x03ffff (0x63e0 bytes) 000671 1778840485.961274 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5006c36a (stale) Mirror 5: 0x5006c36a (stale) Mirror 7: 0x5006c36a (stale) Mirror 8: 0x5006c36a (stale) Mirror 9: 0x5006c36a (stale) Mirror 10: 0x5006c36a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5006c36a (stale) Mirror 5: 0x5006c36a (stale) Mirror 7: 0x5006c36a (stale) Mirror 8: 0x5006c36a (stale) Mirror 9: 0x5006c36a (stale) Mirror 10: 0x5006c36a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000675 1778840486.590574 write 0x03688a thru 0x03ffff (0x9776 bytes) 000676 1778840486.600314 write 0x002b6c thru 0x01252b (0xf9c0 bytes) 000677 1778840486.627494 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5006c36a (stale) Mirror 5: 0x5006c36a (stale) Mirror 7: 0x5006c36a (stale) Mirror 8: 0x5006c36a (stale) Mirror 9: 0x5006c36a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5006c36a (stale) Mirror 5: 0x5006c36a (stale) Mirror 7: 0x5006c36a (stale) Mirror 8: 0x5006c36a (stale) Mirror 9: 0x5006c36a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000678 1778840487.169332 mapread 0x019379 thru 0x02128a (0x7f12 bytes) 000679 1778840487.207125 write 0x02db94 thru 0x03b351 (0xd7be bytes) 000680 1778840487.218428 trunc from 0x040000 to 0x0044b3 (0x3bb4c bytes) 000681 1778840487.242468 mapread 0x002920 thru 0x0044b3 (0x1b94 bytes) 000685 1778840487.255562 mapread 0x00406b thru 0x0044b3 (0x449 bytes) 000688 1778840487.269815 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x44b4): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x44b4): Mirror 4: 0x7562299f (stale) Mirror 5: 0x7562299f (stale) Mirror 7: 0x7562299f (stale) Mirror 8: 0x7562299f (stale) Mirror 9: 0x7562299f (stale) CRC-32 checksum value for chunk [0, 0x44b4): Mirror 1: 0x27b8a4fe Mirror 10: 0x27b8a4fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x44b4: skipped 000690 1778840488.088491 MIRROR_RESYNC 7 mirrors 000691 1778840489.186016 trunc from 0x0044b4 to 0x001097 (0x341c bytes) 000692 1778840489.294159 mapread 0x0009ed thru 0x001097 (0x6ab bytes) 000693 1778840489.305911 mapread 0x00107c thru 0x001097 (0x1c bytes) 000694 1778840489.311529 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1098): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1098): Mirror 4: 0xb16dab5 (stale) Mirror 5: 0xb16dab5 (stale) Mirror 7: 0xb16dab5 (stale) Mirror 8: 0xb16dab5 (stale) Mirror 9: 0xb16dab5 (stale) Mirror 10: 0xb16dab5 (stale) CRC-32 checksum value for chunk [0, 0x1098): Mirror 1: 0xb16dab5 Mirror 11: 0xb16dab5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1098: skipped 000695 1778840490.034994 mapwrite 0x028d73 thru 0x02fa5e (0x6cec bytes) 000696 1778840490.245639 trunc from 0x02fa5f to 0x01a0b0 (0x159ae bytes) 000697 1778840490.268377 mapread 0x00693e thru 0x00ed52 (0x8415 bytes) 000698 1778840490.315793 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a0b1): Mirror 4: 0x1b58898d (stale) Mirror 5: 0x1b58898d (stale) Mirror 7: 0x1b58898d (stale) Mirror 9: 0x1b58898d (stale) Mirror 10: 0x1b58898d (stale) Mirror 11: 0x5de073fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0b1): Mirror 4: 0x1b58898d (stale) Mirror 5: 0x1b58898d (stale) Mirror 7: 0x1b58898d (stale) Mirror 9: 0x1b58898d (stale) Mirror 10: 0x1b58898d (stale) Mirror 11: 0x5de073fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0b1: skipped 000700 1778840490.990453 read 0x00320b thru 0x00d27e (0xa074 bytes) 000702 1778840491.006420 mapread 0x019c8f thru 0x01a0b0 (0x422 bytes) 000703 1778840491.020663 read 0x00a1af thru 0x010927 (0x6779 bytes) 000704 1778840491.038264 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a0b1): Mirror 4: 0x1b58898d (stale) Mirror 5: 0x1b58898d (stale) Mirror 7: 0x1b58898d (stale) Mirror 10: 0x1b58898d (stale) Mirror 11: 0x5de073fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0b1): Mirror 4: 0x1b58898d (stale) Mirror 5: 0x1b58898d (stale) Mirror 7: 0x1b58898d (stale) Mirror 10: 0x1b58898d (stale) Mirror 11: 0x5de073fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0b1: skipped 000708 1778840491.426130 read 0x00ea2e thru 0x01a0b0 (0xb683 bytes) 000711 1778840491.440938 read 0x00556f thru 0x0087ab (0x323d bytes) 000714 1778840491.451684 mapwrite 0x00788a thru 0x013885 (0xbffc bytes) 000716 1778840491.531232 trunc from 0x01a0b1 to 0x000d2b (0x19385 bytes) 000717 1778840491.555214 mapread 0x0008a4 thru 0x000d2b (0x488 bytes) 000719 1778840491.557237 read 0x0009bc thru 0x000d2b (0x370 bytes) 000720 1778840491.558025 trunc from 0x000d2c to 0x0297e7 (0x28abc bytes) 000721 1778840491.607794 mapwrite 0x012729 thru 0x016d5a (0x4632 bytes) 000722 1778840491.725147 read 0x0016ed thru 0x00aae0 (0x93f4 bytes) 000726 1778840491.734928 mapread 0x008f72 thru 0x0148e7 (0xb976 bytes) 000729 1778840491.756821 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x297e8): Mirror 4: 0x17aeaef4 (stale) Mirror 5: 0x17aeaef4 (stale) Mirror 7: 0x17aeaef4 (stale) Mirror 11: 0xaa007c01 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x297e8): Mirror 4: 0x17aeaef4 (stale) Mirror 5: 0x17aeaef4 (stale) Mirror 7: 0x17aeaef4 (stale) Mirror 11: 0xaa007c01 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x297e8: skipped 000730 1778840492.337071 mapread 0x00240e thru 0x00fe19 (0xda0c bytes) 000731 1778840492.371255 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x297e8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x297e8): Mirror 4: 0x17aeaef4 (stale) Mirror 5: 0x17aeaef4 (stale) Mirror 7: 0x17aeaef4 (stale) Mirror 11: 0xaa007c01 (stale) CRC-32 checksum value for chunk [0, 0x297e8): Mirror 1: 0x8ce72dd5 Mirror 12: 0x8ce72dd5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x297e8: skipped 000732 1778840493.117598 mapread 0x01c488 thru 0x027660 (0xb1d9 bytes) 000733 1778840493.162642 read 0x002f49 thru 0x010ded (0xdea5 bytes) 000734 1778840493.198397 mapwrite 0x01f088 thru 0x02e7ed (0xf766 bytes) 000735 1778840493.476078 mapread 0x011a32 thru 0x01ef01 (0xd4d0 bytes) 000736 1778840493.494362 mapwrite 0x01d6a7 thru 0x020391 (0x2ceb bytes) 000737 1778840493.583473 mapread 0x01900b thru 0x01b0ca (0x20c0 bytes) 000740 1778840493.610458 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e7ee): Mirror 5: 0xffb24ad3 (stale) Mirror 7: 0xffb24ad3 (stale) Mirror 11: 0x26e9cf7 (stale) Mirror 12: 0x93941ca5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e7ee): Mirror 5: 0xffb24ad3 (stale) Mirror 7: 0xffb24ad3 (stale) Mirror 11: 0x26e9cf7 (stale) Mirror 12: 0x93941ca5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e7ee: skipped 000741 1778840494.126258 write 0x00408b thru 0x011dc9 (0xdd3f bytes) 000743 1778840494.138897 mapwrite 0x01de96 thru 0x029cfa (0xbe65 bytes) 000746 1778840494.266598 read 0x01f33a thru 0x02e7ed (0xf4b4 bytes) 000747 1778840494.267625 trunc from 0x02e7ee to 0x01d0a1 (0x1174c bytes) 000750 1778840494.334573 read 0x0197ea thru 0x01d0a1 (0x38b8 bytes) 000751 1778840494.337147 mapwrite 0x0012aa thru 0x001785 (0x4dc bytes) 000752 1778840494.410425 mapwrite 0x022721 thru 0x02fd44 (0xd624 bytes) 000754 1778840494.485603 write 0x01affb thru 0x021d09 (0x6d0f bytes) 000756 1778840494.488194 write 0x00cf76 thru 0x01c67e (0xf709 bytes) 000757 1778840494.490823 read 0x017e20 thru 0x01ca97 (0x4c78 bytes) 000759 1778840494.492703 trunc from 0x02fd45 to 0x035db8 (0x6074 bytes) 000760 1778840494.533975 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35db9): Mirror 7: 0xdbfcb4a (stale) Mirror 11: 0x6a9c0c4b (stale) Mirror 12: 0xb5d9d9a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35db9): Mirror 7: 0xdbfcb4a (stale) Mirror 11: 0x6a9c0c4b (stale) Mirror 12: 0xb5d9d9a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35db9: skipped 000761 1778840494.925768 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35db9): Mirror 11: 0x6a9c0c4b (stale) Mirror 12: 0xb5d9d9a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35db9): Mirror 11: 0x6a9c0c4b (stale) Mirror 12: 0xb5d9d9a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35db9: skipped 000762 1778840495.247679 trunc from 0x035db9 to 0x01b67e (0x1a73a bytes) 000763 1778840495.267097 write 0x01877b thru 0x023cf1 (0xb577 bytes) 000766 1778840495.283244 trunc from 0x023cf2 to 0x01b5fa (0x86f7 bytes) 000767 1778840495.309711 write 0x0206e0 thru 0x02bba1 (0xb4c2 bytes) 000768 1778840495.313192 read 0x01ee45 thru 0x021425 (0x25e1 bytes) 000769 1778840495.321233 trunc from 0x02bba2 to 0x00278a (0x29417 bytes) 000771 1778840495.343381 mapwrite 0x005836 thru 0x00d568 (0x7d33 bytes) 000772 1778840495.439420 mapwrite 0x0059ca thru 0x01024e (0xa885 bytes) 000774 1778840495.555410 write 0x02184c thru 0x0290d8 (0x788d bytes) 000775 1778840495.560983 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x290d9): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x290d9): Mirror 11: 0x7f11c84d (stale) Mirror 12: 0x37183571 (stale) CRC-32 checksum value for chunk [0, 0x290d9): Mirror 1: 0xe2bbcb92 Mirror 13: 0xe2bbcb92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x290d9: skipped 000776 1778840496.528631 trunc from 0x0290d9 to 0x01c76b (0xc96d bytes) 000778 1778840496.639114 write 0x012da8 thru 0x02185b (0xeab4 bytes) 000779 1778840496.652246 mapread 0x019ae8 thru 0x021825 (0x7d3e bytes) 000780 1778840496.654857 trunc from 0x02185c to 0x032d80 (0x11525 bytes) 000782 1778840496.684981 trunc from 0x032d81 to 0x0296e8 (0x9698 bytes) 000783 1778840496.713034 mapread 0x010a08 thru 0x01d183 (0xc77c bytes) 000784 1778840496.734204 mapwrite 0x03c9f3 thru 0x03eb4b (0x2159 bytes) 000785 1778840496.904412 trunc from 0x03eb4c to 0x00abc2 (0x33f89 bytes) 000786 1778840496.933100 trunc from 0x00abc3 to 0x02100b (0x16449 bytes) 000789 1778840496.967192 mapwrite 0x03d21b thru 0x03ffff (0x2de5 bytes) 000790 1778840497.051089 trunc from 0x040000 to 0x0011b3 (0x3ee4c bytes) 000791 1778840497.068239 mapwrite 0x027545 thru 0x0371fe (0xfcba bytes) 000792 1778840497.155749 read 0x01286e thru 0x01d3f1 (0xab84 bytes) 000793 1778840497.166164 mapread 0x02ab01 thru 0x02ed12 (0x4212 bytes) 000794 1778840497.167826 read 0x004fa6 thru 0x00aa6e (0x5ac9 bytes) 000795 1778840497.177508 mapwrite 0x003835 thru 0x00debf (0xa68b bytes) 000797 1778840497.226541 write 0x028d6e thru 0x03008c (0x731f bytes) 000798 1778840497.229073 write 0x03bef5 thru 0x03ffff (0x410b bytes) 000801 1778840497.232962 read 0x02b0c9 thru 0x02e320 (0x3258 bytes) 000802 1778840497.233808 mapwrite 0x01a2a4 thru 0x01a62d (0x38a bytes) 000805 1778840497.275644 mapread 0x003b94 thru 0x0118b5 (0xdd22 bytes) 000806 1778840497.283060 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x3d1be6b5 (stale) Mirror 13: 0x594c868d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x3d1be6b5 (stale) Mirror 13: 0x594c868d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000808 1778840497.624043 write 0x037827 thru 0x03ffff (0x87d9 bytes) 000809 1778840497.632314 read 0x03947f thru 0x03d0b7 (0x3c39 bytes) 000810 1778840497.633106 trunc from 0x040000 to 0x03a8cc (0x5733 bytes) 000811 1778840497.665018 read 0x011878 thru 0x01c17b (0xa904 bytes) 000812 1778840497.687189 MIRROR_RESYNC 3 mirrors 000813 1778840498.740752 mapwrite 0x0090f4 thru 0x00ec6a (0x5b77 bytes) 000814 1778840498.913912 mapwrite 0x00db18 thru 0x01cc34 (0xf11d bytes) 000816 1778840499.009218 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a8cd): Mirror 12: 0xe4690f50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a8cd): Mirror 12: 0xe4690f50 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a8cd: skipped 000817 1778840499.522603 mapread 0x01d7f3 thru 0x01e735 (0xf43 bytes) 000819 1778840499.546436 mapwrite 0x00a424 thru 0x00b7a2 (0x137f bytes) 000821 1778840499.602018 trunc from 0x03a8cd to 0x01ab21 (0x1fdab bytes) 000822 1778840499.626138 MIRROR_RESYNC 2 mirrors 000823 1778840500.301685 read 0x008d24 thru 0x00a736 (0x1a13 bytes) 000826 1778840500.308661 read 0x01448e thru 0x01ab21 (0x6694 bytes) 000827 1778840500.317971 mapread 0x00c8b7 thru 0x017b01 (0xb24b bytes) 000828 1778840500.321039 read 0x014603 thru 0x0191be (0x4bbc bytes) 000829 1778840500.321811 read 0x005637 thru 0x010619 (0xafe3 bytes) 000830 1778840500.322634 write 0x0393c7 thru 0x03ffff (0x6c39 bytes) 000832 1778840500.388631 write 0x01c2c7 thru 0x024556 (0x8290 bytes) truncating to largest ever: 0x3ead0 000834 1778840500.402817 trunc from 0x040000 to 0x03eacf (0x1530 bytes) 000835 1778840500.428971 trunc from 0x03ead0 to 0x018bf1 (0x25ede bytes) 000836 1778840500.446895 mapread 0x011969 thru 0x018bf1 (0x7289 bytes) 000837 1778840500.449846 mapread 0x00a81b thru 0x01301c (0x8802 bytes) 000839 1778840500.458025 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x18bf2): Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18bf2): Mirror 12: 0x18e45370 (stale) CRC-32 checksum value for chunk [0, 0x18bf2): Mirror 1: 0x18e45370 Mirror 13: 0x18e45370 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18bf2: skipped 000841 1778840501.232124 mapwrite 0x01d8f0 thru 0x01e266 (0x977 bytes) 000842 1778840501.425841 mapread 0x006f2d thru 0x0159ba (0xea8e bytes) 000843 1778840501.453930 read 0x016663 thru 0x01e266 (0x7c04 bytes) 000844 1778840501.460272 write 0x0289cc thru 0x02f714 (0x6d49 bytes) 000845 1778840501.473925 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f715): Mirror 13: 0x8ecbe703 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f715): Mirror 13: 0x8ecbe703 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f715: skipped 000847 1778840501.822789 trunc from 0x02f715 to 0x01b121 (0x145f3 bytes) 000848 1778840501.864829 read 0x005410 thru 0x010ff6 (0xbbe7 bytes) 000849 1778840501.883091 mapwrite 0x00fa82 thru 0x011024 (0x15a3 bytes) 000851 1778840501.946426 mapwrite 0x033fb7 thru 0x034cc1 (0xd0b bytes) 000852 1778840502.057090 mapwrite 0x01ef98 thru 0x025466 (0x64cf bytes) 000853 1778840502.133086 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) 000854 1778840502.278481 read 0x018094 thru 0x026f84 (0xeef1 bytes) 000856 1778840502.297528 mapread 0x019df9 thru 0x01b5b0 (0x17b8 bytes) 000857 1778840502.300257 trunc from 0x034cc2 to 0x00e964 (0x2635d bytes) 000859 1778840502.325422 read 0x00ab7a thru 0x00e964 (0x3deb bytes) 000864 1778840502.345926 trunc from 0x00e965 to 0x028b05 (0x1a1a1 bytes) 000868 1778840502.379661 write 0x02357a thru 0x0237c6 (0x24d bytes) 000869 1778840502.415979 mapread 0x015f94 thru 0x01b1ae (0x521b bytes) 000874 1778840502.427556 write 0x012fac thru 0x016834 (0x3889 bytes) 000876 1778840502.432778 read 0x00571b thru 0x011933 (0xc219 bytes) 000878 1778840502.435562 write 0x039de8 thru 0x03ffff (0x6218 bytes) 000880 1778840502.443618 trunc from 0x040000 to 0x02e821 (0x117de bytes) 000881 1778840502.476058 write 0x0329cc thru 0x03f23d (0xc872 bytes) 000883 1778840502.479927 trunc from 0x03f23e to 0x01acd3 (0x2456a bytes) 000886 1778840502.497582 trunc from 0x01acd4 to 0x03ad09 (0x20036 bytes) 000889 1778840502.510984 mapread 0x009fc1 thru 0x00f969 (0x59a9 bytes) 000891 1778840502.512666 mapwrite 0x00160d thru 0x0106a2 (0xf096 bytes) 000894 1778840502.559653 mapread 0x00781d thru 0x00d23d (0x5a21 bytes) 000895 1778840502.561489 mapwrite 0x00b76a thru 0x00fda9 (0x4640 bytes) 000896 1778840502.603050 trunc from 0x03ad0a to 0x005c9b (0x3506e bytes) 000897 1778840502.636612 write 0x03cf0b thru 0x03ffff (0x30f5 bytes) 000900 1778840502.641914 trunc from 0x040000 to 0x029e5d (0x161a2 bytes) 000904 1778840502.679022 mapwrite 0x02f4f7 thru 0x03a795 (0xb29f bytes) 000905 1778840502.779167 MIRROR_EXTEND to 2 mirrors 000906 1778840503.687214 write 0x02e0d6 thru 0x03d7d3 (0xf6fe bytes) 000908 1778840503.820183 mapwrite 0x020ad5 thru 0x02cb07 (0xc033 bytes) 000910 1778840503.913667 write 0x028177 thru 0x036729 (0xe5b3 bytes) 000912 1778840503.917904 trunc from 0x03d7d4 to 0x023ad0 (0x19d03 bytes) 000914 1778840503.949857 mapwrite 0x03a714 thru 0x03ffff (0x58ec bytes) 000915 1778840504.032386 mapwrite 0x00531a thru 0x00b393 (0x607a bytes) 000916 1778840504.109688 mapread 0x00e2fa thru 0x01c8b7 (0xe5be bytes) 000918 1778840504.121599 trunc from 0x040000 to 0x008145 (0x37eba bytes) 000919 1778840504.139986 mapread 0x002558 thru 0x008145 (0x5bee bytes) 000921 1778840504.143049 write 0x02cf78 thru 0x034b9b (0x7c24 bytes) 000922 1778840504.152627 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) 000923 1778840504.359311 write 0x0061f2 thru 0x0092eb (0x30fa bytes) 000925 1778840504.375025 write 0x006222 thru 0x0092fc (0x30db bytes) 000931 1778840504.387348 write 0x000616 thru 0x0075a0 (0x6f8b bytes) 000932 1778840504.398667 write 0x008e24 thru 0x017972 (0xeb4f bytes) 000933 1778840504.405622 mapwrite 0x023771 thru 0x0269cb (0x325b bytes) 000934 1778840504.468033 trunc from 0x034b9c to 0x02626e (0xe92d bytes) 000936 1778840504.496640 mapwrite 0x01f522 thru 0x02c0ca (0xcba9 bytes) 000937 1778840504.567871 mapread 0x01c30c thru 0x022b90 (0x6885 bytes) 000938 1778840504.570430 mapread 0x0195c4 thru 0x027f31 (0xe96e bytes) 000939 1778840504.577078 write 0x01b8aa thru 0x01d12b (0x1882 bytes) 000940 1778840504.580880 mapwrite 0x023891 thru 0x027eea (0x465a bytes) 000941 1778840504.615897 trunc from 0x02c0cb to 0x0292e6 (0x2de4 bytes) 000942 1778840504.659060 MIRROR_EXTEND to 2 mirrors 000943 1778840505.277484 mapwrite 0x03c8a5 thru 0x03ffff (0x375b bytes) 000946 1778840505.457475 mapwrite 0x022cab thru 0x02573b (0x2a91 bytes) 000947 1778840505.539157 read 0x01c55e thru 0x02965c (0xd0ff bytes) 000950 1778840505.541184 write 0x030f53 thru 0x035202 (0x42b0 bytes) 000951 1778840505.556649 write 0x02d471 thru 0x031fa9 (0x4b39 bytes) 000953 1778840505.558559 read 0x01fd71 thru 0x02259a (0x282a bytes) 000956 1778840505.561093 trunc from 0x040000 to 0x0290b8 (0x16f47 bytes) 000957 1778840505.593387 mapwrite 0x037eee thru 0x03ffff (0x8112 bytes) 000958 1778840505.753170 mapread 0x00e085 thru 0x019f02 (0xbe7e bytes) 000959 1778840505.758696 write 0x015cbc thru 0x01b258 (0x559d bytes) 000960 1778840505.767014 trunc from 0x040000 to 0x0346d9 (0xb926 bytes) 000962 1778840505.790299 trunc from 0x0346da to 0x030078 (0x4661 bytes) 000965 1778840505.812634 mapread 0x010f9c thru 0x014156 (0x31bb bytes) 000967 1778840505.817024 write 0x03d063 thru 0x03ffff (0x2f9d bytes) 000968 1778840505.819621 mapread 0x000ac8 thru 0x00e0e7 (0xd620 bytes) 000970 1778840505.833682 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d6:0x0]: Invalid argument (22) 000973 1778840506.019033 mapread 0x0275e9 thru 0x029103 (0x1b1b bytes) 000974 1778840506.045396 mapwrite 0x038ed8 thru 0x03b3f2 (0x251b bytes) 000975 1778840506.098271 read 0x037541 thru 0x03ffff (0x8abf bytes) 000979 1778840506.104686 write 0x01cb66 thru 0x01d1bf (0x65a bytes) 000981 1778840506.119017 trunc from 0x040000 to 0x000aea (0x3f515 bytes) 000982 1778840506.131011 mapwrite 0x01fc14 thru 0x02f4d4 (0xf8c1 bytes) 000983 1778840506.217652 mapwrite 0x002fcb thru 0x0113a7 (0xe3dd bytes) 000984 1778840506.292935 MIRROR_EXTEND to 2 mirrors 000986 1778840506.894898 mapwrite 0x0013e0 thru 0x007aa4 (0x66c5 bytes) 000987 1778840507.067494 mapread 0x01c81a thru 0x029b10 (0xd2f7 bytes) 000991 1778840507.092493 mapread 0x0160a8 thru 0x01cd66 (0x6cbf bytes) 000992 1778840507.096880 write 0x03a363 thru 0x03f85d (0x54fb bytes) 000993 1778840507.100997 mapread 0x00b42c thru 0x01b400 (0xffd5 bytes) 000995 1778840507.128900 mapwrite 0x016655 thru 0x01da22 (0x73ce bytes) 000996 1778840507.202198 read 0x00b1a8 thru 0x00b934 (0x78d bytes) 000997 1778840507.203970 read 0x03641b thru 0x03d5dd (0x71c3 bytes) 000998 1778840507.224182 mapwrite 0x02fc7f thru 0x0363e8 (0x676a bytes) 000999 1778840507.290844 write 0x0295ad thru 0x02cd57 (0x37ab bytes) 001000 1778840507.292102 mapread 0x028bea thru 0x030bfb (0x8012 bytes) All operations completed A-OK! PASS 70a (104s) == sanity-flr test 71: check mirror extend parent fid ==== 06:21:52 (1778840512) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 06:21:55 (1778840515) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211938 s, 9.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116092 s, 9.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x329:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 06:22:03 (1778840523) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5388544 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8896864 .. done Extending file size to 8266048 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8682208 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8969696 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9050496 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9286560 .. Waiting 56884 56885 56887 56888 56889 done 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) 192.168.201.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:23:20 (1778840600) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5470656 .. Extending file size to 4807648 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8592576 .. Extending file size to 5780608 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8239136 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8364160 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9102720 .. done Waiting 58572 58574 58575 58577 58578 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) 192.168.201.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:24:39 (1778840679) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2816 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 2816 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0958056 s, 21.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.27485 s, 7.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf8758573 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:24:56 (1778840696) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129319 s, 16.2 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 06:25:02 (1778840702) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22a:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x340:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x340:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x344:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:25:09 (1778840709) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.196094 s, 10.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x345:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x346:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:25:15 (1778840715) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138765 s, 15.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x347:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x347:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:25:22 (1778840722) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21256 s, 9.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x345:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x345:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:25:51 (1778840751) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg123-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100194 s, 10.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:26:49 (1778840809) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg123-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175948 s, 11.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x349:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:27:45 (1778840865) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg123-server mds-ost sync done. Creating new pool test_204f oleg123-server: Pool lustre.test_204f created Adding targets to pool oleg123-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg123-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.275773 s, 7.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x34a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg123-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg123-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg123-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:29:00 (1778840940) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:29:06 (1778840946) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:29:13 (1778840953) oleg123-server: Pool lustre.test_206 created Adding targets to pool oleg123-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg123-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg123-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg123-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg123-server: Pool lustre.test_206 destroyed Waiting 90s for 'foo' PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:29:35 (1778840975) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819936 s, 12.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758796 s, 13.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2ce52bd Mirror 3: 0x2ce52bd PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:29:45 (1778840985) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:29:49 (1778840989) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:29:52 (1778840992) Starting client: oleg123-client.virtnet: -o user_xattr,flock 192.168.201.123@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0734192 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x970000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1630000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1630000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1630000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xed0000 Read this (4 bytes): BBBB 192.168.201.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg123-client.virtnet /mnt/lustre2 (opts:) PASS 209a (32s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:30:24 (1778841024) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0163255 s, 502 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:30:31 (1778841031) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439533 s, 23.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x24f lmm_fid: [0x200000404:0x24f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x24f lmm_fid: [0x200000404:0x24f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x358:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834511 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-15 06:30:32.000000000 -0400 Modify: 2026-05-15 06:30:32.000000000 -0400 Change: 2026-05-15 06:30:33.000000000 -0400 Birth: 2026-05-15 06:30:31.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:30:38 (1778841038) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:30:41 (1778841041) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.383813 s, 107 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0462304 s, 88.6 kB/s size after second write -rw-r--r-- 1 root root 4096 May 15 06:30 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x251:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 15 06:30 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:30:49 (1778841049) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:30:56 (1778841056) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1768 sec =========== 06:31:04 (1778841064) === sanity-flr: start cleanup 06:31:05 (1778841065) === === sanity-flr: finish cleanup 06:31:10 (1778841070) ===