-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 07:13:36 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg230-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 07:13:56 (1787829236) === oleg230-client.virtnet: executing check_config_client /mnt/lustre oleg230-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg230-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d22d11bc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d22d11bc000.idle_timeout=debug disable quota as required oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 07:14:41 (1787829281) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7618] == sanity-flr test 0a: lfs mirror create with -N option == 07:14:44 (1787829284) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:15:02 (1787829302) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:15:06 (1787829306) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:15:10 (1787829310) PASS 0d (32s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:15:42 (1787829342) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:15:46 (1787829346) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 07:15:49 (1787829349) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d22d11bc000.stats=clear osc.lustre-OST0001-osc-ffff8d22d11bc000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:15:58 (1787829358) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:16:10 (1787829370) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:16:19 (1787829379) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:16:23 (1787829383) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:16:32 (1787829392) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 07:16:42 (1787829402) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:16:49 (1787829409) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:16:53 (1787829413) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:17:00 (1787829420) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:17:10 (1787829430) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:17:17 (1787829437) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 2.55611 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186466 s, 5.6 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:17:31 (1787829451) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.522358 s, 24.1 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:17:42 (1787829462) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:17:51 (1787829471) Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d22d11bc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d22d11bc000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg230-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d22d11bc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d22d11bc000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (134s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:20:05 (1787829605) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.561467 s, 16.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d22d11bc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d22d11bc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:20:47 (1787829647) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d22d11bc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d22d11bc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d22d11bc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d22d11bc000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:22:11 (1787829731) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.7743 s, 11.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 07:22 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.612s user 0m0.001s sys 0m0.206s Stopping /mnt/lustre-ost1 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d22d11bc000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d22d11bc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.251s user 0m0.001s sys 0m0.189s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg230-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg230-server oleg230-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d22d11bc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d22d11bc000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.348s user 0m0.001s sys 0m0.168s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg230-server: oleg230-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg230-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg230-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d22d11bc000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d22d11bc000.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:23:51 (1787829831) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:23:55 (1787829835) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:23:58 (1787829838) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 07:24:02 (1787829842) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 07:24:10 (1787829850) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.729745 s, 17.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:24:24 (1787829864) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288523 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457587 s, 22.9 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:24:32 (1787829872) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:24:40 (1787829880) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00123222 s, 831 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00115494 s, 887 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0010846 s, 944 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000964574 s, 1.1 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 07:24:51 (1787829891) create test files with size 15754303 .. f18b62a94202c7267d6a65a21b6af66a - 0dc9327126b555f129cde7fb275060bc - b64733eac4beaa5a2e95a02598dbea71 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 07:25:13 (1787829913) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:25:31 (1787829931) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:25:38 (1787829938) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158034 s, 6.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272677 s, 38.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 07:25:49 (1787829949) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121121 s, 17.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.429685 s, 9.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216847 s, 9.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 07:26:07 (1787829967) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 07:26:10 (1787829970) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:26:14 (1787829974) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316909 s, 33.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 07:26:22 (1787829982) llite.lustre-ffff8d22c603d800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (6s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 07:26:29 (1787829989) llite.lustre-ffff8d22c603d800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288984 s, 36.3 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420008 s, 25.0 MB/s **verify components PASS 43d (10s) == sanity-flr test 44a: lfs mirror split check =========== 07:26:39 (1787829999) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 07:26:43 (1787830003) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xca:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xca:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 07:26:54 (1787830014) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.344501 s, 30.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 07:27:08 (1787830028) llite.lustre-ffff8d22c3669800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.350578 s, 29.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg230-client.virtnet /mnt/lustre (opts:) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8d22d0ecd800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 07:27:23 (1787830043) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 07:28:05 (1787830085) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (51s) == sanity-flr test 47: Verify mirror obj alloc =========== 07:28:56 (1787830136) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 07:29:00 (1787830140) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.329061 s, 9.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.229582 s, 13.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8437da7b (nosync,stale) Mirror 2: 0x8437da7b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1a29c48940aef9c36972f4531790aef5 - mirror 1 checksum: b8d809ac64253c16649b59d6c9fa11cc - mirror 2 checksum: 1a29c48940aef9c36972f4531790aef5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T07:29:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T07:29:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1a29c48940aef9c36972f4531790aef5 - mirror 1 checksum: b8d809ac64253c16649b59d6c9fa11cc - mirror 2 checksum: b8d809ac64253c16649b59d6c9fa11cc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T07:29:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T07:29:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 48 (15s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 07:29:17 (1787830157) Failing mds1 on oleg230-server Stopping /mnt/lustre-mds1 (opts:) on oleg230-server 07:29:23 (1787830163) shut down facet: mds1 facet_host: oleg230-server facet_failover_host: oleg230-server Failover mds1 to oleg230-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg230-server: oleg230-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg230-client: oleg230-server: ssh exited with exit code 1 Started lustre-MDT0000 07:29:57 (1787830197) targets are mounted 07:29:57 (1787830197) facet_failover done oleg230-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260061 s, 40.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } PASS 50A (62s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 07:30:19 (1787830219) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 07:30:23 (1787830223) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 07:30:27 (1787830227) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00595426 s, 2.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 07:30:35 (1787830235) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 07:30:39 (1787830239) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0106808 s, 383 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0510298 s, 80.3 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 07:30:47 (1787830247) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (26s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 07:31:12 (1787830272) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 07:31:42 (1787830302) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 07:32:07 (1787830327) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 07:32:32 (1787830352) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 07:32:41 (1787830361) mirror create pid 55591 mirror split pid 55592 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6991: 55592 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6991: 55592 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55591 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55592 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 07:33:49 (1787830429) Seed set to 431 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x39382 000001 1787830432.136460 trunc from 00000000 to 0x039381 (0x39382 bytes) 000003 1787830432.178551 write 0x034795 thru 0x03ffff (0xb86b bytes) 000005 1787830432.192343 mapwrite 0x0065cf thru 0x00e343 (0x7d75 bytes) 000009 1787830432.319375 mapread 0x00c06c thru 0x01853f (0xc4d4 bytes) 000010 1787830432.344806 MIRROR_EXTEND to 2 mirrors 000011 1787830433.171820 read 0x005860 thru 0x011024 (0xb7c5 bytes) 000013 1787830433.185755 read 0x021c4c thru 0x030a3f (0xedf4 bytes) 000014 1787830433.198586 write 0x00cf89 thru 0x01640b (0x9483 bytes) 000015 1787830433.287944 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000016 1787830433.437045 mapread 0x030dd5 thru 0x0343e9 (0x3615 bytes) 000017 1787830433.475451 write 0x03db72 thru 0x03ffff (0x248e bytes) 000018 1787830433.483472 mapwrite 0x0164c7 thru 0x01d8a3 (0x73dd bytes) 000019 1787830433.546480 trunc from 0x040000 to 0x026fa2 (0x1905d bytes) 000020 1787830433.589838 mapwrite 0x01a123 thru 0x0247c1 (0xa69f bytes) 000024 1787830433.648481 write 0x01222b thru 0x01c53c (0xa312 bytes) 000026 1787830433.657542 MIRROR_EXTEND to 2 mirrors 000027 1787830434.299322 MIRROR_RESYNC 2 mirrors 000028 1787830434.669278 read 0x01d8a3 thru 0x02411e (0x687c bytes) 000029 1787830434.677526 mapread 0x007065 thru 0x00f233 (0x81cf bytes) 000030 1787830434.695271 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000033 1787830434.819921 trunc from 0x026fa3 to 0x024390 (0x2c12 bytes) 000035 1787830434.845456 write 0x0383de thru 0x03cdf7 (0x4a1a bytes) 000036 1787830434.846653 read 0x021f8c thru 0x023651 (0x16c6 bytes) 000037 1787830434.852464 mapwrite 0x01ab27 thru 0x028093 (0xd56d bytes) 000040 1787830434.935235 MIRROR_EXTEND to 2 mirrors 000041 1787830435.737695 MIRROR_EXTEND to 3 mirrors 000042 1787830436.581016 read 0x0164ce thru 0x01df89 (0x7abc bytes) 000044 1787830436.594894 mapwrite 0x019aee thru 0x01f51e (0x5a31 bytes) 000045 1787830436.924470 write 0x00435f thru 0x011bcd (0xd86f bytes) 000047 1787830436.941901 trunc from 0x03cdf8 to 0x0243c3 (0x18a34 bytes) 000049 1787830436.976526 read 0x022515 thru 0x0243c3 (0x1eaf bytes) 000051 1787830436.990511 write 0x000968 thru 0x009254 (0x88ed bytes) 000053 1787830436.994085 write 0x02d35e thru 0x0305c8 (0x326b bytes) 000054 1787830436.995884 mapwrite 0x005830 thru 0x005cb8 (0x489 bytes) 000055 1787830437.045238 mapread 0x013cea thru 0x01dcdb (0x9ff2 bytes) 000057 1787830437.048008 trunc from 0x0305c9 to 0x0342ae (0x3ce6 bytes) 000062 1787830437.067975 write 0x00d88f thru 0x01d421 (0xfb93 bytes) 000064 1787830437.070747 read 0x0190ff thru 0x01b736 (0x2638 bytes) 000065 1787830437.073453 mapread 0x013ae3 thru 0x016885 (0x2da3 bytes) 000066 1787830437.074642 mapwrite 0x01790e thru 0x026c8c (0xf37f bytes) 000067 1787830437.179046 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x342af): Mirror 3: 0x9b09436f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342af): Mirror 3: 0x9b09436f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342af: skipped 000069 1787830437.576782 read 0x00f67a thru 0x015bd7 (0x655e bytes) 000070 1787830437.587944 read 0x00b2b6 thru 0x011fbe (0x6d09 bytes) 000073 1787830437.600486 trunc from 0x0342af to 0x0301c1 (0x40ed bytes) 000075 1787830437.626332 MIRROR_RESYNC 2 mirrors 000078 1787830438.570517 write 0x00c8bc thru 0x01253f (0x5c84 bytes) 000079 1787830438.691349 trunc from 0x0301c2 to 0x01a8f5 (0x158cc bytes) 000081 1787830438.731823 mapwrite 0x01f4c8 thru 0x0249e0 (0x5519 bytes) 000082 1787830438.848601 MIRROR_RESYNC 2 mirrors 000083 1787830439.809829 mapread 0x016398 thru 0x017b80 (0x17e9 bytes) 000084 1787830439.825358 write 0x03b484 thru 0x03e58e (0x310b bytes) 000087 1787830439.888830 read 0x036196 thru 0x039d20 (0x3b8b bytes) 000092 1787830439.898577 write 0x01e144 thru 0x027a87 (0x9944 bytes) 000094 1787830439.913776 MIRROR_RESYNC 2 mirrors 000096 1787830440.577275 mapwrite 0x0110fc thru 0x013a16 (0x291b bytes) 000097 1787830440.736753 write 0x009bcd thru 0x017c00 (0xe034 bytes) 000098 1787830440.749894 mapread 0x00b781 thru 0x01314b (0x79cb bytes) 000099 1787830440.756256 trunc from 0x03e58f to 0x02d406 (0x11188 bytes) 000100 1787830440.809070 read 0x021909 thru 0x0224c4 (0xbbc bytes) 000101 1787830440.821014 mapwrite 0x01a57d thru 0x01ef28 (0x49ac bytes) 000102 1787830440.910005 mapread 0x014edb thru 0x018b3e (0x3c64 bytes) 000103 1787830440.910740 mapwrite 0x0035bf thru 0x00f3f8 (0xbe3a bytes) 000104 1787830440.969363 mapread 0x005796 thru 0x00c971 (0x71dc bytes) 000105 1787830440.977257 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000107 1787830441.260824 write 0x00a3c7 thru 0x0161d6 (0xbe10 bytes) 000110 1787830441.290288 write 0x02ea02 thru 0x0383bc (0x99bb bytes) 000111 1787830441.292245 mapread 0x0339e9 thru 0x0383bc (0x49d4 bytes) 000113 1787830441.303539 mapread 0x0147d6 thru 0x015513 (0xd3e bytes) 000114 1787830441.309409 read 0x02db80 thru 0x0383bc (0xa83d bytes) 000115 1787830441.332865 mapwrite 0x03ee38 thru 0x03ffff (0x11c8 bytes) 000116 1787830441.505587 write 0x03b909 thru 0x03c07c (0x774 bytes) 000119 1787830441.527232 mapwrite 0x006b0e thru 0x00c7e2 (0x5cd5 bytes) 000120 1787830441.625676 mapwrite 0x022313 thru 0x02d67b (0xb369 bytes) 000121 1787830441.755990 mapread 0x039e35 thru 0x03ffff (0x61cb bytes) 000122 1787830441.765036 read 0x0128c8 thru 0x0167c1 (0x3efa bytes) 000124 1787830441.768388 trunc from 0x040000 to 0x02ca50 (0x135af bytes) 000125 1787830441.786478 read 0x011dad thru 0x01730c (0x5560 bytes) 000126 1787830441.793900 read 0x02c295 thru 0x02ca50 (0x7bc bytes) 000127 1787830441.794867 mapread 0x017324 thru 0x02529c (0xdf79 bytes) 000130 1787830441.800194 MIRROR_EXTEND to 2 mirrors 000131 1787830442.657631 read 0x006db0 thru 0x00da7f (0x6cd0 bytes) 000132 1787830442.666737 mapread 0x00f7d2 thru 0x010d5c (0x158b bytes) 000133 1787830442.684970 MIRROR_RESYNC 2 mirrors 000136 1787830443.050918 mapwrite 0x00da42 thru 0x019dd0 (0xc38f bytes) 000138 1787830443.226887 mapwrite 0x0011bb thru 0x009cfe (0x8b44 bytes) 000139 1787830443.314754 mapread 0x00457e thru 0x0065ee (0x2071 bytes) 000143 1787830443.320891 write 0x00fee7 thru 0x011635 (0x174f bytes) 000144 1787830443.322874 mapread 0x029397 thru 0x02ca50 (0x36ba bytes) 000149 1787830443.342656 read 0x00d776 thru 0x01587b (0x8106 bytes) 000151 1787830443.344632 read 0x02525c thru 0x02aaa8 (0x584d bytes) truncating to largest ever: 0x3a070 000152 1787830443.346429 trunc from 0x02ca51 to 0x03a06f (0xd61f bytes) 000153 1787830443.363830 mapread 0x00ed13 thru 0x0103ec (0x16da bytes) 000155 1787830443.368602 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a070): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a070): Mirror 2: 0xbc6e8b33 (stale) CRC-32 checksum value for chunk [0, 0x3a070): Mirror 1: 0xccc09667 Mirror 3: 0xccc09667 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a070: skipped 000157 1787830444.946078 trunc from 0x03a070 to 0x010e78 (0x291f7 bytes) 000163 1787830445.104326 mapwrite 0x0162a0 thru 0x01c2f1 (0x6052 bytes) 000166 1787830445.229969 mapread 0x0079ef thru 0x012cc2 (0xb2d4 bytes) 000167 1787830445.242554 read 0x00eaa3 thru 0x01be45 (0xd3a3 bytes) 000168 1787830445.243903 mapread 0x01a0b4 thru 0x01af4e (0xe9b bytes) 000169 1787830445.245766 mapread 0x006afd thru 0x0150d0 (0xe5d4 bytes) 000172 1787830445.247884 mapread 0x011813 thru 0x01be24 (0xa612 bytes) 000173 1787830445.250435 mapread 0x01bcd2 thru 0x01c2f1 (0x620 bytes) 000175 1787830445.252629 mapread 0x00ce1d thru 0x00d7b8 (0x99c bytes) 000176 1787830445.254572 trunc from 0x01c2f2 to 0x01ce10 (0xb1f bytes) 000177 1787830445.271492 write 0x00e886 thru 0x0145a1 (0x5d1c bytes) 000179 1787830445.274953 mapwrite 0x034283 thru 0x03ffff (0xbd7d bytes) 000180 1787830445.374862 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfe7b841d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfe7b841d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000181 1787830445.708015 read 0x00c12e thru 0x010cdb (0x4bae bytes) 000182 1787830445.717817 mapwrite 0x01d013 thru 0x029845 (0xc833 bytes) 000183 1787830445.812223 write 0x030a1d thru 0x037b09 (0x70ed bytes) 000184 1787830445.820176 read 0x032fbf thru 0x03ffff (0xd041 bytes) 000186 1787830445.834628 mapread 0x01103d thru 0x01ecb0 (0xdc74 bytes) 000188 1787830445.851942 write 0x000d07 thru 0x003325 (0x261f bytes) 000190 1787830445.854139 read 0x028003 thru 0x037bff (0xfbfd bytes) truncating to largest ever: 0x3e355 000191 1787830445.857023 trunc from 0x040000 to 0x03e354 (0x1cab bytes) 000192 1787830445.880393 read 0x03d521 thru 0x03e354 (0xe34 bytes) 000193 1787830445.882153 mapread 0x00c0d4 thru 0x01afbc (0xeee9 bytes) 000195 1787830445.883939 write 0x02622c thru 0x031a88 (0xb85d bytes) 000197 1787830445.886068 mapread 0x03d2c2 thru 0x03e354 (0x1093 bytes) 000198 1787830445.886997 mapwrite 0x03656f thru 0x03ffff (0x9a91 bytes) 000200 1787830446.042448 trunc from 0x040000 to 0x020017 (0x1ffe8 bytes) 000202 1787830446.076210 mapread 0x018c7f thru 0x01be7d (0x31ff bytes) 000204 1787830446.079284 trunc from 0x020018 to 0x035131 (0x1511a bytes) 000206 1787830446.102918 write 0x03a21e thru 0x03ffff (0x5de2 bytes) 000207 1787830446.104712 mapread 0x009435 thru 0x01565d (0xc229 bytes) 000209 1787830446.113431 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000210 1787830446.462296 read 0x02a03e thru 0x02b467 (0x142a bytes) 000212 1787830446.469759 mapwrite 0x001363 thru 0x00f749 (0xe3e7 bytes) 000214 1787830446.548588 mapread 0x00b1dc thru 0x00f618 (0x443d bytes) 000215 1787830446.549468 write 0x02ccde thru 0x03b870 (0xeb93 bytes) 000216 1787830446.558427 read 0x0254dc thru 0x0301e4 (0xad09 bytes) 000218 1787830446.565518 read 0x00f524 thru 0x01ee70 (0xf94d bytes) 000219 1787830446.566247 mapread 0x00a368 thru 0x00ac00 (0x899 bytes) 000220 1787830446.567309 mapread 0x001d74 thru 0x00dc1e (0xbeab bytes) 000221 1787830446.567959 mapread 0x025706 thru 0x02fba1 (0xa49c bytes) 000222 1787830446.571206 mapread 0x03aeb9 thru 0x03ffff (0x5147 bytes) 000228 1787830446.577033 trunc from 0x040000 to 0x01e39d (0x21c62 bytes) 000231 1787830446.622964 trunc from 0x01e39e to 0x0196ad (0x4cf0 bytes) 000232 1787830446.656288 read 0x0095e5 thru 0x010eb0 (0x78cc bytes) 000233 1787830446.657703 mapread 0x014d90 thru 0x0180c2 (0x3333 bytes) 000235 1787830446.659735 mapwrite 0x009c8b thru 0x01092e (0x6ca4 bytes) 000236 1787830446.748372 mapread 0x00dc7e thru 0x018a9d (0xae20 bytes) 000238 1787830446.749968 trunc from 0x0196ae to 0x010fe9 (0x86c4 bytes) 000239 1787830446.778219 mapwrite 0x02e96f thru 0x03935c (0xa9ee bytes) 000240 1787830446.852728 trunc from 0x03935d to 0x00eaaf (0x2a8ad bytes) 000241 1787830446.883849 trunc from 0x00eab0 to 0x0141af (0x5700 bytes) 000242 1787830446.920305 MIRROR_EXTEND to 2 mirrors 000244 1787830448.413049 read 0x008006 thru 0x012604 (0xa5ff bytes) 000246 1787830448.443993 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000248 1787830448.592700 mapread 0x0088c1 thru 0x0141af (0xb8ef bytes) 000249 1787830448.617622 read 0x0100d6 thru 0x0141af (0x40da bytes) 000251 1787830448.619182 mapwrite 0x001960 thru 0x0044b0 (0x2b51 bytes) 000252 1787830448.689490 write 0x00cd47 thru 0x01a0bc (0xd376 bytes) 000253 1787830448.695733 MIRROR_EXTEND to 2 mirrors 000255 1787830449.497376 read 0x000f1e thru 0x005264 (0x4347 bytes) 000259 1787830449.521928 read 0x004aeb thru 0x011fa5 (0xd4bb bytes) 000260 1787830449.538208 trunc from 0x01a0bd to 0x01bd2c (0x1c70 bytes) 000263 1787830449.619382 mapwrite 0x0069d9 thru 0x0073dc (0xa04 bytes) 000264 1787830449.671328 trunc from 0x01bd2d to 0x01f3ab (0x367f bytes) 000265 1787830449.695965 mapread 0x01b63f thru 0x01f3ab (0x3d6d bytes) 000266 1787830449.704858 write 0x008beb thru 0x01624d (0xd663 bytes) 000267 1787830449.716879 trunc from 0x01f3ac to 0x028ef7 (0x9b4c bytes) 000268 1787830449.738264 mapwrite 0x0213b9 thru 0x0224a0 (0x10e8 bytes) 000271 1787830449.793982 trunc from 0x028ef8 to 0x00ad24 (0x1e1d3 bytes) 000273 1787830449.822010 write 0x021e7a thru 0x023617 (0x179e bytes) 000274 1787830449.832593 MIRROR_RESYNC 2 mirrors 000275 1787830450.539000 mapwrite 0x00c880 thru 0x00f3a5 (0x2b26 bytes) 000276 1787830450.669443 mapwrite 0x012f68 thru 0x01371a (0x7b3 bytes) 000277 1787830450.707623 trunc from 0x023618 to 0x03b268 (0x17c51 bytes) 000279 1787830450.734880 mapread 0x01e7e3 thru 0x024df2 (0x6610 bytes) 000280 1787830450.746964 write 0x03a0a9 thru 0x03ffff (0x5f57 bytes) 000282 1787830450.753505 mapwrite 0x03e592 thru 0x03ffff (0x1a6e bytes) 000283 1787830450.791386 write 0x00775d thru 0x011ae9 (0xa38d bytes) 000286 1787830450.793213 read 0x007f08 thru 0x012e34 (0xaf2d bytes) 000287 1787830450.793655 mapread 0x00b3be thru 0x0190bd (0xdd00 bytes) 000289 1787830450.800325 mapwrite 0x0285ff thru 0x02ea77 (0x6479 bytes) 000294 1787830450.853181 MIRROR_RESYNC 2 mirrors 000295 1787830451.616510 read 0x008c8c thru 0x0180d7 (0xf44c bytes) 000299 1787830451.635424 mapwrite 0x03d90b thru 0x03ffff (0x26f5 bytes) 000301 1787830451.807621 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9a4a6c47 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaad18593 Mirror 3: 0xaad18593 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000303 1787830452.783748 write 0x00ded4 thru 0x00e1ef (0x31c bytes) 000305 1787830452.886897 write 0x03c1ff thru 0x03d57b (0x137d bytes) 000307 1787830452.909562 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9a4a6c47 (stale) Mirror 3: 0xaad18593 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc8a9264f Mirror 4: 0xc8a9264f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000309 1787830454.137949 mapwrite 0x019d84 thru 0x01af8a (0x1207 bytes) 000310 1787830454.421942 write 0x01fbc6 thru 0x02cd6e (0xd1a9 bytes) 000311 1787830454.429827 read 0x00a0bb thru 0x00b483 (0x13c9 bytes) 000312 1787830454.440667 mapwrite 0x01a793 thru 0x022561 (0x7dcf bytes) 000313 1787830454.508592 trunc from 0x040000 to 0x02ec16 (0x113e9 bytes) 000314 1787830454.544440 MIRROR_RESYNC 4 mirrors truncating to largest ever: 0x3eb23 000315 1787830455.716698 trunc from 0x02ec17 to 0x03eb22 (0xff0c bytes) 000316 1787830455.761842 mapread 0x008297 thru 0x00fd4c (0x7ab6 bytes) 000317 1787830455.784424 trunc from 0x03eb23 to 0x033242 (0xb8e0 bytes) 000319 1787830455.812895 MIRROR_RESYNC 4 mirrors 000320 1787830457.153211 mapwrite 0x006c82 thru 0x00cb06 (0x5e85 bytes) 000321 1787830457.400545 mapwrite 0x01e7c2 thru 0x026193 (0x79d2 bytes) 000322 1787830457.581024 mapwrite 0x008d51 thru 0x013201 (0xa4b1 bytes) 000326 1787830457.748748 MIRROR_RESYNC 4 mirrors 000328 1787830459.378863 write 0x025423 thru 0x02d5db (0x81b9 bytes) 000329 1787830459.521623 MIRROR_RESYNC 4 mirrors 000330 1787830460.778242 write 0x03a698 thru 0x03acf8 (0x661 bytes) 000331 1787830460.890496 read 0x021658 thru 0x024d83 (0x372c bytes) 000333 1787830460.899211 read 0x03834d thru 0x03acf8 (0x29ac bytes) 000334 1787830460.939837 write 0x00d6db thru 0x0154dd (0x7e03 bytes) 000337 1787830460.948108 write 0x039e57 thru 0x03ffff (0x61a9 bytes) 000338 1787830460.949304 mapwrite 0x012177 thru 0x017f69 (0x5df3 bytes) 000345 1787830461.154645 read 0x02e660 thru 0x0307e1 (0x2182 bytes) 000347 1787830461.157262 read 0x005dc1 thru 0x0073ea (0x162a bytes) 000350 1787830461.159951 mapwrite 0x017ec1 thru 0x0199f4 (0x1b34 bytes) 000351 1787830461.234360 mapwrite 0x0355e4 thru 0x03cbf9 (0x7616 bytes) 000353 1787830461.317820 mapread 0x014240 thru 0x014922 (0x6e3 bytes) 000354 1787830461.319425 mapwrite 0x01d805 thru 0x02cad7 (0xf2d3 bytes) 000356 1787830461.464530 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa273d327 (stale) Mirror 3: 0xa273d327 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa273d327 (stale) Mirror 3: 0xa273d327 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000357 1787830461.984583 trunc from 0x040000 to 0x026481 (0x19b7e bytes) 000358 1787830462.000162 mapwrite 0x01927c thru 0x01bbcc (0x2951 bytes) 000360 1787830462.123453 write 0x000d24 thru 0x009bfc (0x8ed9 bytes) 000363 1787830462.163411 MIRROR_RESYNC 3 mirrors 000365 1787830463.196964 mapread 0x023aa0 thru 0x026481 (0x29e2 bytes) 000370 1787830463.217468 mapwrite 0x022c8a thru 0x02c8e5 (0x9c5c bytes) 000372 1787830463.438457 mapread 0x00d6f2 thru 0x0176f3 (0xa002 bytes) 000373 1787830463.466251 mapread 0x00e4d7 thru 0x01aa39 (0xc563 bytes) 000374 1787830463.467543 write 0x023ad5 thru 0x024eb5 (0x13e1 bytes) 000377 1787830463.470491 trunc from 0x02c8e6 to 0x02a9b3 (0x1f32 bytes) 000378 1787830463.502268 mapread 0x018bc7 thru 0x019b46 (0xf80 bytes) 000380 1787830463.505482 mapwrite 0x03b77f thru 0x03ffff (0x4881 bytes) 000381 1787830463.590625 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf31ca4ab (stale) Mirror 3: 0xf31ca4ab (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd0e6947a Mirror 4: 0xd0e6947a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000382 1787830464.381202 mapwrite 0x039a3d thru 0x03ffff (0x65c3 bytes) 000383 1787830464.658047 trunc from 0x040000 to 0x017e53 (0x281ac bytes) 000384 1787830464.699371 mapread 0x00bdb4 thru 0x017e53 (0xc0a0 bytes) 000385 1787830464.720702 read 0x00bea2 thru 0x00fc3e (0x3d9d bytes) 000386 1787830464.721690 mapwrite 0x00b9f4 thru 0x00dfe4 (0x25f1 bytes) 000390 1787830464.765613 mapwrite 0x016fb5 thru 0x024b60 (0xdbac bytes) 000391 1787830464.866416 read 0x009697 thru 0x016074 (0xc9de bytes) 000392 1787830464.867446 read 0x0239f5 thru 0x024b60 (0x116c bytes) 000393 1787830464.868725 mapread 0x00ed33 thru 0x01cf1d (0xe1eb bytes) 000396 1787830464.878038 MIRROR_RESYNC 4 mirrors 000397 1787830466.055151 trunc from 0x024b61 to 0x005a63 (0x1f0fd bytes) 000401 1787830466.209174 mapread 0x005525 thru 0x005a63 (0x53f bytes) 000403 1787830466.232919 mapwrite 0x01b0ab thru 0x01c478 (0x13ce bytes) 000404 1787830466.401232 write 0x018cb9 thru 0x021b74 (0x8ebc bytes) 000405 1787830466.410640 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x21b75): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21b75): Mirror 2: 0x642aefec (stale) Mirror 3: 0x642aefec (stale) Mirror 4: 0x642aefec (stale) CRC-32 checksum value for chunk [0, 0x21b75): Mirror 1: 0x2f7991a8 Mirror 5: 0x2f7991a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21b75: skipped 000406 1787830467.418432 mapwrite 0x010c0d thru 0x0112cd (0x6c1 bytes) 000408 1787830467.563745 read 0x0217e4 thru 0x021b74 (0x391 bytes) 000409 1787830467.568506 mapread 0x01bd20 thru 0x01ec5a (0x2f3b bytes) 000411 1787830467.576355 write 0x00f050 thru 0x00f483 (0x434 bytes) 000415 1787830467.578827 read 0x00bb7c thru 0x0104ec (0x4971 bytes) 000417 1787830467.579877 read 0x012ecd thru 0x020601 (0xd735 bytes) 000420 1787830467.581399 read 0x00d55a thru 0x00da9f (0x546 bytes) 000421 1787830467.581952 mapwrite 0x001561 thru 0x00a2d9 (0x8d79 bytes) 000422 1787830467.619612 mapread 0x00ee83 thru 0x01daa0 (0xec1e bytes) 000423 1787830467.620543 read 0x005da3 thru 0x010a6d (0xaccb bytes) 000424 1787830467.621827 mapread 0x01df9d thru 0x021b74 (0x3bd8 bytes) 000426 1787830467.630203 write 0x000e5b thru 0x0010a2 (0x248 bytes) 000427 1787830467.638920 read 0x018a6f thru 0x021b74 (0x9106 bytes) 000429 1787830467.655496 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x21b75): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21b75): Mirror 2: 0x642aefec (stale) Mirror 3: 0x642aefec (stale) Mirror 4: 0x642aefec (stale) Mirror 5: 0x2f7991a8 (stale) CRC-32 checksum value for chunk [0, 0x21b75): Mirror 1: 0x70fa9fad Mirror 6: 0x70fa9fad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21b75: skipped 000431 1787830469.109362 mapwrite 0x009a1a thru 0x009c1c (0x203 bytes) 000433 1787830469.263692 write 0x03f12f thru 0x03ffff (0xed1 bytes) 000434 1787830469.277526 mapread 0x010fcb thru 0x01c3f7 (0xb42d bytes) 000436 1787830469.315060 read 0x03c0d4 thru 0x03ffff (0x3f2c bytes) 000437 1787830469.326554 write 0x02a474 thru 0x030f53 (0x6ae0 bytes) 000439 1787830469.356945 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe860446f (stale) Mirror 3: 0xe860446f (stale) Mirror 4: 0xe860446f (stale) Mirror 5: 0xa4f7ace1 (stale) Mirror 6: 0x41084133 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf917251d Mirror 7: 0xf917251d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000441 1787830470.294728 mapread 0x016b39 thru 0x0196f1 (0x2bb9 bytes) 000442 1787830470.317915 mapread 0x03e26a thru 0x03ffff (0x1d96 bytes) 000443 1787830470.333902 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe860446f (stale) Mirror 3: 0xe860446f (stale) Mirror 4: 0xe860446f (stale) Mirror 5: 0xa4f7ace1 (stale) Mirror 6: 0x41084133 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf917251d Mirror 7: 0xf917251d Mirror 8: 0xf917251d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000445 1787830471.498969 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe860446f (stale) Mirror 3: 0xe860446f (stale) Mirror 4: 0xe860446f (stale) Mirror 6: 0x41084133 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe860446f (stale) Mirror 3: 0xe860446f (stale) Mirror 4: 0xe860446f (stale) Mirror 6: 0x41084133 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf917251d Mirror 7: 0xf917251d Mirror 8: 0xf917251d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000446 1787830472.462631 mapwrite 0x010833 thru 0x010a12 (0x1e0 bytes) 000448 1787830472.618033 write 0x01961c thru 0x02015e (0x6b43 bytes) 000451 1787830472.632680 read 0x026bfb thru 0x0313fa (0xa800 bytes) 000452 1787830472.643381 trunc from 0x040000 to 0x03cea0 (0x315f bytes) 000453 1787830472.665152 trunc from 0x03cea1 to 0x01bd13 (0x2118d bytes) 000454 1787830472.689028 MIRROR_RESYNC 7 mirrors 000455 1787830474.388625 MIRROR_RESYNC 7 mirrors 000456 1787830475.018030 write 0x03cab9 thru 0x03ffff (0x3547 bytes) 000460 1787830475.174621 mapread 0x0042d3 thru 0x00ab37 (0x6865 bytes) 000462 1787830475.202729 mapread 0x0287e8 thru 0x0325cb (0x9de4 bytes) 000464 1787830475.246421 read 0x03d8c0 thru 0x03ffff (0x2740 bytes) 000465 1787830475.266770 MIRROR_RESYNC 7 mirrors 000466 1787830477.403020 trunc from 0x040000 to 0x02cce3 (0x1331c bytes) truncating to largest ever: 0x3f61b 000469 1787830477.552978 trunc from 0x02cce4 to 0x03f61a (0x12937 bytes) 000470 1787830477.606720 write 0x0214fe thru 0x02eaff (0xd602 bytes) 000471 1787830477.630401 trunc from 0x03f61b to 0x01eb1e (0x20afc bytes) 000473 1787830477.672846 trunc from 0x01eb1f to 0x0039ea (0x1b134 bytes) 000474 1787830477.714534 trunc from 0x0039eb to 0x025ae1 (0x220f7 bytes) 000475 1787830477.749257 write 0x0017be thru 0x006bb5 (0x53f8 bytes) 000476 1787830477.762393 trunc from 0x025ae2 to 0x01eff6 (0x6aeb bytes) 000478 1787830477.806378 read 0x00b753 thru 0x00e926 (0x31d4 bytes) 000479 1787830477.823277 write 0x03522e thru 0x03acb7 (0x5a8a bytes) 000481 1787830477.827187 trunc from 0x03acb8 to 0x02cdef (0xdec8 bytes) 000482 1787830477.852354 trunc from 0x02cdf0 to 0x005811 (0x275de bytes) 000485 1787830477.898202 mapread 0x004462 thru 0x005811 (0x13b0 bytes) 000486 1787830477.919520 MIRROR_RESYNC 7 mirrors 000487 1787830479.463437 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000489 1787830479.865470 read 0x0033da thru 0x005811 (0x2438 bytes) 000490 1787830479.879538 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000493 1787830480.390083 write 0x0049a7 thru 0x0099bb (0x5015 bytes) 000494 1787830480.521843 read 0x00259a thru 0x0099bb (0x7422 bytes) 000495 1787830480.533833 read 0x002d47 thru 0x0099bb (0x6c75 bytes) 000496 1787830480.536021 write 0x017c3c thru 0x024673 (0xca38 bytes) 000499 1787830480.549577 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24674): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24674): Mirror 3: 0x4e244bb4 (stale) Mirror 6: 0x4e244bb4 (stale) Mirror 7: 0x4e244bb4 (stale) Mirror 8: 0x4e244bb4 (stale) CRC-32 checksum value for chunk [0, 0x24674): Mirror 1: 0x3d51502a Mirror 9: 0x3d51502a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24674: skipped 000500 1787830481.558452 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24674): Mirror 3: 0x4e244bb4 (stale) Mirror 6: 0x4e244bb4 (stale) Mirror 7: 0x4e244bb4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24674): Mirror 3: 0x4e244bb4 (stale) Mirror 6: 0x4e244bb4 (stale) Mirror 7: 0x4e244bb4 (stale) CRC-32 checksum value for chunk [0, 0x24674): Mirror 1: 0x3d51502a Mirror 9: 0x3d51502a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24674: skipped 000501 1787830482.406574 write 0x03447d thru 0x03e751 (0xa2d5 bytes) 000502 1787830482.644484 trunc from 0x03e752 to 0x026a33 (0x17d1e bytes) 000503 1787830482.747019 mapread 0x0061d0 thru 0x008261 (0x2092 bytes) 000504 1787830482.798480 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x26a34): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a34): Mirror 3: 0xb77c9964 (stale) Mirror 6: 0xb77c9964 (stale) Mirror 7: 0xb77c9964 (stale) Mirror 9: 0x1a3d0dbb (stale) CRC-32 checksum value for chunk [0, 0x26a34): Mirror 1: 0x1a3d0dbb Mirror 10: 0x1a3d0dbb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a34: skipped 000505 1787830483.896373 trunc from 0x026a34 to 0x02dfa1 (0x756e bytes) 000508 1787830484.022974 read 0x02132b thru 0x0297ee (0x84c4 bytes) 000509 1787830484.038270 mapread 0x017131 thru 0x01f78c (0x865c bytes) 000510 1787830484.077882 trunc from 0x02dfa2 to 0x023b3e (0xa463 bytes) 000511 1787830484.117237 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23b3f): Mirror 3: 0xe1a25ffa (stale) Mirror 7: 0xe1a25ffa (stale) Mirror 9: 0x164f8a19 (stale) Mirror 10: 0x164f8a19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23b3f): Mirror 3: 0xe1a25ffa (stale) Mirror 7: 0xe1a25ffa (stale) Mirror 9: 0x164f8a19 (stale) Mirror 10: 0x164f8a19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23b3f: skipped 000514 1787830484.734961 write 0x01cc1e thru 0x0215c1 (0x49a4 bytes) 000515 1787830484.748936 mapwrite 0x0361bc thru 0x03ffff (0x9e44 bytes) 000517 1787830484.895803 write 0x03d20b thru 0x03d3ac (0x1a2 bytes) 000518 1787830484.896889 read 0x010bb1 thru 0x015b56 (0x4fa6 bytes) 000519 1787830484.908530 mapwrite 0x0329cb thru 0x036b31 (0x4167 bytes) 000522 1787830484.965620 mapread 0x0102e2 thru 0x01bbd6 (0xb8f5 bytes) 000524 1787830484.981680 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x5d55da9c (stale) Mirror 7: 0x5d55da9c (stale) Mirror 9: 0x87177c27 (stale) Mirror 10: 0x87177c27 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x84f426a6 Mirror 11: 0x84f426a6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000526 1787830486.103665 mapwrite 0x00d58a thru 0x01b2b3 (0xdd2a bytes) 000527 1787830486.295575 mapread 0x031590 thru 0x035a2a (0x449b bytes) 000529 1787830486.308620 mapread 0x00455c thru 0x00c20b (0x7cb0 bytes) 000531 1787830486.319498 write 0x021f0e thru 0x02c75e (0xa851 bytes) 000532 1787830486.326224 write 0x03e317 thru 0x03ffff (0x1ce9 bytes) 000533 1787830486.333776 write 0x014e94 thru 0x0154bc (0x629 bytes) 000534 1787830486.339563 MIRROR_RESYNC 6 mirrors 000535 1787830487.602521 mapread 0x01bbde thru 0x01cd5e (0x1181 bytes) 000537 1787830487.614876 write 0x00f4c4 thru 0x01b832 (0xc36f bytes) 000538 1787830487.685378 mapread 0x008184 thru 0x00ef4f (0x6dcc bytes) 000539 1787830487.700797 MIRROR_RESYNC 6 mirrors 000541 1787830489.016289 mapread 0x017db6 thru 0x020b10 (0x8d5b bytes) 000544 1787830489.052346 write 0x024cb7 thru 0x026491 (0x17db bytes) 000545 1787830489.182904 mapwrite 0x01a92f thru 0x0210b9 (0x678b bytes) 000546 1787830489.245979 write 0x02fd9a thru 0x038862 (0x8ac9 bytes) 000547 1787830489.257935 mapread 0x008788 thru 0x011821 (0x909a bytes) 000550 1787830489.276564 mapread 0x038aed thru 0x03ffff (0x7513 bytes) 000551 1787830489.290487 mapwrite 0x02db45 thru 0x03c27b (0xe737 bytes) 000554 1787830489.362719 mapwrite 0x02cedd thru 0x03135e (0x4482 bytes) 000555 1787830489.405573 mapread 0x033b22 thru 0x03950c (0x59eb bytes) 000560 1787830489.414561 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe4a7eb5a (stale) Mirror 7: 0xe4a7eb5a (stale) Mirror 9: 0xe4a7eb5a (stale) Mirror 10: 0xe4a7eb5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe4a7eb5a (stale) Mirror 7: 0xe4a7eb5a (stale) Mirror 9: 0xe4a7eb5a (stale) Mirror 10: 0xe4a7eb5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000561 1787830489.942951 mapwrite 0x02819f thru 0x034e49 (0xccab bytes) 000564 1787830490.041427 read 0x0197ba thru 0x02378b (0x9fd2 bytes) 000568 1787830490.072922 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe4a7eb5a (stale) Mirror 7: 0xe4a7eb5a (stale) Mirror 9: 0xe4a7eb5a (stale) Mirror 10: 0xe4a7eb5a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x64cf23dc Mirror 11: 0x64cf23dc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000569 1787830491.317669 mapread 0x014949 thru 0x018431 (0x3ae9 bytes) 000570 1787830491.333339 trunc from 0x040000 to 0x021047 (0x1efb8 bytes) 000571 1787830491.405590 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x21048): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21048): Mirror 3: 0xcf29f5fb (stale) Mirror 7: 0xcf29f5fb (stale) Mirror 9: 0xcf29f5fb (stale) Mirror 10: 0xcf29f5fb (stale) Mirror 11: 0x2a9c331f (stale) CRC-32 checksum value for chunk [0, 0x21048): Mirror 1: 0x2a9c331f Mirror 12: 0x2a9c331f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21048: skipped 000572 1787830492.825023 mapread 0x01fbc6 thru 0x021047 (0x1482 bytes) 000575 1787830492.875412 mapwrite 0x002200 thru 0x00aa37 (0x8838 bytes) 000576 1787830493.085042 read 0x00159b thru 0x00eaa9 (0xd50f bytes) 000577 1787830493.085779 trunc from 0x021048 to 0x01f222 (0x1e25 bytes) 000579 1787830493.114477 mapwrite 0x0334df thru 0x03a25d (0x6d7f bytes) 000580 1787830493.196506 trunc from 0x03a25e to 0x01d89d (0x1c9c0 bytes) 000581 1787830493.223599 read 0x01d683 thru 0x01d89d (0x21b bytes) 000585 1787830493.238840 trunc from 0x01d89e to 0x01d32e (0x56f bytes) 000587 1787830493.256875 mapread 0x0180b9 thru 0x01a079 (0x1fc1 bytes) 000589 1787830493.261363 mapread 0x015607 thru 0x01d32e (0x7d28 bytes) 000592 1787830493.265475 mapwrite 0x01e9ad thru 0x02191a (0x2f6e bytes) 000593 1787830493.329960 write 0x030697 thru 0x037f3b (0x78a5 bytes) 000594 1787830493.339399 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x37f3c): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f3c): Mirror 3: 0x5659ce27 (stale) Mirror 7: 0x5659ce27 (stale) Mirror 9: 0x5659ce27 (stale) Mirror 10: 0x5659ce27 (stale) Mirror 11: 0xdb5bc519 (stale) Mirror 12: 0x4e8ba27f (stale) CRC-32 checksum value for chunk [0, 0x37f3c): Mirror 1: 0xe22d6521 Mirror 13: 0xe22d6521 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f3c: skipped 000595 1787830494.327445 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x37f3c): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37f3c): Mirror 3: 0x5659ce27 (stale) Mirror 7: 0x5659ce27 (stale) Mirror 9: 0x5659ce27 (stale) Mirror 10: 0x5659ce27 (stale) Mirror 11: 0xdb5bc519 (stale) Mirror 12: 0x4e8ba27f (stale) CRC-32 checksum value for chunk [0, 0x37f3c): Mirror 1: 0xe22d6521 Mirror 13: 0xe22d6521 Mirror 14: 0xe22d6521 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37f3c: skipped 000596 1787830495.799752 write 0x0376d3 thru 0x03ffff (0x892d bytes) 000597 1787830495.919169 read 0x0291c3 thru 0x02b44e (0x228c bytes) 000598 1787830495.929457 write 0x0243a8 thru 0x031def (0xda48 bytes) 000599 1787830495.952694 trunc from 0x040000 to 0x02b0a8 (0x14f57 bytes) 000601 1787830495.992061 trunc from 0x02b0a9 to 0x039ea0 (0xedf8 bytes) 000602 1787830496.021771 mapwrite 0x01473d thru 0x0197e6 (0x50aa bytes) 000603 1787830496.146170 mapread 0x02200e thru 0x02deef (0xbee2 bytes) 000605 1787830496.189865 mapwrite 0x02da3e thru 0x039bc8 (0xc18b bytes) 000607 1787830496.320385 trunc from 0x039ea1 to 0x00ac7e (0x2f222 bytes) 000609 1787830496.374552 MIRROR_RESYNC 9 mirrors 000610 1787830498.240918 mapwrite 0x005f1f thru 0x00a748 (0x482a bytes) 000611 1787830498.401639 MIRROR_RESYNC 9 mirrors 000613 1787830499.772148 trunc from 0x00ac7f to 0x032779 (0x27afb bytes) 000614 1787830499.888568 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3277a): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3277a): Mirror 3: 0x178774e7 (stale) Mirror 7: 0x178774e7 (stale) Mirror 9: 0x178774e7 (stale) Mirror 10: 0x178774e7 (stale) Mirror 11: 0x178774e7 (stale) Mirror 12: 0x178774e7 (stale) Mirror 13: 0x178774e7 (stale) Mirror 14: 0x178774e7 (stale) CRC-32 checksum value for chunk [0, 0x3277a): Mirror 1: 0x178774e7 Mirror 15: 0x178774e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3277a: skipped 000616 1787830501.124537 write 0x0274e4 thru 0x0322a2 (0xadbf bytes) 000617 1787830501.246780 mapwrite 0x038aab thru 0x03ffff (0x7555 bytes) 000619 1787830501.366687 write 0x0288c4 thru 0x037b5b (0xf298 bytes) 000622 1787830501.377453 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xbd97febe (stale) Mirror 7: 0xbd97febe (stale) Mirror 9: 0xbd97febe (stale) Mirror 10: 0xbd97febe (stale) Mirror 11: 0xbd97febe (stale) Mirror 12: 0xbd97febe (stale) Mirror 13: 0xbd97febe (stale) Mirror 14: 0xbd97febe (stale) Mirror 15: 0xbd97febe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc461d0a Mirror 16: 0xc461d0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000623 1787830503.210181 mapread 0x018195 thru 0x026ab6 (0xe922 bytes) 000624 1787830503.258528 mapread 0x021b00 thru 0x021fe7 (0x4e8 bytes) 000625 1787830503.263368 read 0x03023d thru 0x03b37e (0xb142 bytes) 000626 1787830503.280732 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbd97febe (stale) Mirror 9: 0xbd97febe (stale) Mirror 10: 0xbd97febe (stale) Mirror 11: 0xbd97febe (stale) Mirror 12: 0xbd97febe (stale) Mirror 13: 0xbd97febe (stale) Mirror 14: 0xbd97febe (stale) Mirror 15: 0xbd97febe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xbd97febe (stale) Mirror 9: 0xbd97febe (stale) Mirror 10: 0xbd97febe (stale) Mirror 11: 0xbd97febe (stale) Mirror 12: 0xbd97febe (stale) Mirror 13: 0xbd97febe (stale) Mirror 14: 0xbd97febe (stale) Mirror 15: 0xbd97febe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc461d0a Mirror 16: 0xc461d0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000627 1787830504.283659 read 0x02da9d thru 0x035e00 (0x8364 bytes) 000629 1787830504.316762 write 0x006182 thru 0x00db29 (0x79a8 bytes) 000630 1787830504.490929 read 0x008dc3 thru 0x00d8b2 (0x4af0 bytes) 000632 1787830504.492375 trunc from 0x040000 to 0x031c36 (0xe3c9 bytes) 000634 1787830504.524497 mapwrite 0x0097b0 thru 0x00dedd (0x472e bytes) 000636 1787830504.578960 read 0x02057d thru 0x024e30 (0x48b4 bytes) 000637 1787830504.592683 read 0x01561a thru 0x01d806 (0x81ed bytes) 000638 1787830504.593696 write 0x0020f6 thru 0x00e252 (0xc15d bytes) 000641 1787830504.597320 trunc from 0x031c37 to 0x02e2b0 (0x3986 bytes) 000642 1787830504.620019 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2e2b1): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e2b1): Mirror 7: 0xce295634 (stale) Mirror 9: 0xce295634 (stale) Mirror 10: 0xce295634 (stale) Mirror 11: 0xce295634 (stale) Mirror 12: 0xce295634 (stale) Mirror 13: 0xce295634 (stale) Mirror 14: 0xce295634 (stale) Mirror 15: 0xce295634 (stale) Mirror 16: 0xe99c4326 (stale) CRC-32 checksum value for chunk [0, 0x2e2b1): Mirror 1: 0xbea33da9 Mirror 17: 0xbea33da9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e2b1: skipped 000643 1787830506.182908 mapread 0x010a86 thru 0x01f5b4 (0xeb2f bytes) 000645 1787830506.233998 mapwrite 0x037afa thru 0x03ffff (0x8506 bytes) 000646 1787830506.526647 read 0x00b641 thru 0x016715 (0xb0d5 bytes) 000647 1787830506.554768 trunc from 0x040000 to 0x03263a (0xd9c5 bytes) 000648 1787830506.605977 mapread 0x02d3b9 thru 0x02d5f2 (0x23a bytes) 000650 1787830506.626984 mapread 0x015750 thru 0x01902b (0x38dc bytes) 000651 1787830506.656150 mapwrite 0x0267f9 thru 0x02e505 (0x7d0d bytes) 000652 1787830506.734779 read 0x013cbb thru 0x01fd8d (0xc0d3 bytes) 000654 1787830506.736905 trunc from 0x03263b to 0x01ef1f (0x1371b bytes) 000655 1787830506.773814 MIRROR_RESYNC 11 mirrors 000661 1787830508.948863 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000662 1787830509.320924 trunc from 0x01ef20 to 0x0038a6 (0x1b679 bytes) 000666 1787830509.374941 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38a7): Mirror 7: 0xf1dd375e (stale) Mirror 9: 0xf1dd375e (stale) Mirror 10: 0xf1dd375e (stale) Mirror 12: 0xf1dd375e (stale) Mirror 13: 0xf1dd375e (stale) Mirror 14: 0xf1dd375e (stale) Mirror 16: 0xf1dd375e (stale) Mirror 17: 0xf1dd375e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a7): Mirror 7: 0xf1dd375e (stale) Mirror 9: 0xf1dd375e (stale) Mirror 10: 0xf1dd375e (stale) Mirror 12: 0xf1dd375e (stale) Mirror 13: 0xf1dd375e (stale) Mirror 14: 0xf1dd375e (stale) Mirror 16: 0xf1dd375e (stale) Mirror 17: 0xf1dd375e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a7: skipped 000667 1787830509.776271 trunc from 0x0038a7 to 0x025407 (0x21b61 bytes) 000668 1787830509.820278 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x25408): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25408): Mirror 7: 0xcca8f88 (stale) Mirror 9: 0xcca8f88 (stale) Mirror 10: 0xcca8f88 (stale) Mirror 12: 0xcca8f88 (stale) Mirror 13: 0xcca8f88 (stale) Mirror 14: 0xcca8f88 (stale) Mirror 16: 0xcca8f88 (stale) Mirror 17: 0xcca8f88 (stale) CRC-32 checksum value for chunk [0, 0x25408): Mirror 1: 0xc1ddb0f9 Mirror 18: 0xc1ddb0f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25408: skipped 000669 1787830511.079043 mapread 0x01e21e thru 0x025407 (0x71ea bytes) 000670 1787830511.105909 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25408): Mirror 7: 0xcca8f88 (stale) Mirror 10: 0xcca8f88 (stale) Mirror 12: 0xcca8f88 (stale) Mirror 13: 0xcca8f88 (stale) Mirror 14: 0xcca8f88 (stale) Mirror 16: 0xcca8f88 (stale) Mirror 17: 0xcca8f88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25408): Mirror 7: 0xcca8f88 (stale) Mirror 10: 0xcca8f88 (stale) Mirror 12: 0xcca8f88 (stale) Mirror 13: 0xcca8f88 (stale) Mirror 14: 0xcca8f88 (stale) Mirror 16: 0xcca8f88 (stale) Mirror 17: 0xcca8f88 (stale) CRC-32 checksum value for chunk [0, 0x25408): Mirror 1: 0xc1ddb0f9 Mirror 18: 0xc1ddb0f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25408: skipped 000671 1787830512.210647 mapwrite 0x008239 thru 0x00a096 (0x1e5e bytes) 000672 1787830512.300427 write 0x03fba3 thru 0x03ffff (0x45d bytes) 000673 1787830512.301321 trunc from 0x040000 to 0x00e34c (0x31cb3 bytes) 000674 1787830512.327013 MIRROR_RESYNC 9 mirrors 000675 1787830514.326136 MIRROR_EXTEND to 10 mirrors 000678 1787830515.448846 mapwrite 0x01fd66 thru 0x023e3a (0x40d5 bytes) 000679 1787830515.623475 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23e3b): Mirror 7: 0x420f4153 (stale) Mirror 10: 0x420f4153 (stale) Mirror 12: 0x420f4153 (stale) Mirror 13: 0x420f4153 (stale) Mirror 14: 0x420f4153 (stale) Mirror 16: 0x420f4153 (stale) Mirror 17: 0x420f4153 (stale) Mirror 19: 0x420f4153 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23e3b): Mirror 7: 0x420f4153 (stale) Mirror 10: 0x420f4153 (stale) Mirror 12: 0x420f4153 (stale) Mirror 13: 0x420f4153 (stale) Mirror 14: 0x420f4153 (stale) Mirror 16: 0x420f4153 (stale) Mirror 17: 0x420f4153 (stale) Mirror 19: 0x420f4153 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23e3b: skipped 000681 1787830516.316910 mapwrite 0x014597 thru 0x017261 (0x2ccb bytes) 000682 1787830516.385916 write 0x0395fe thru 0x03ffff (0x6a02 bytes) 000684 1787830516.387916 read 0x030110 thru 0x032e14 (0x2d05 bytes) 000685 1787830516.396174 write 0x025a63 thru 0x02f7ca (0x9d68 bytes) 000686 1787830516.406667 trunc from 0x040000 to 0x022894 (0x1d76b bytes) 000688 1787830516.425872 mapread 0x0043d3 thru 0x005842 (0x1470 bytes) 000689 1787830516.438541 read 0x0082d3 thru 0x00f5aa (0x72d8 bytes) 000691 1787830516.439872 mapread 0x020ec6 thru 0x022894 (0x19cf bytes) 000692 1787830516.454706 MIRROR_RESYNC 9 mirrors 000693 1787830517.803490 trunc from 0x022895 to 0x000bce (0x21cc6 bytes) 000695 1787830517.884528 mapwrite 0x00a751 thru 0x00d663 (0x2f13 bytes) 000696 1787830517.980705 write 0x02e9d8 thru 0x03c1a4 (0xd7cd bytes) 000697 1787830517.982915 mapread 0x007b26 thru 0x01471c (0xcbf7 bytes) 000700 1787830518.005951 read 0x02b87a thru 0x036709 (0xae90 bytes) 000701 1787830518.011658 read 0x035ea8 thru 0x036554 (0x6ad bytes) 000702 1787830518.012255 mapread 0x020f43 thru 0x025199 (0x4257 bytes) 000703 1787830518.028634 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c1a5): Mirror 7: 0x2caca141 (stale) Mirror 12: 0x2caca141 (stale) Mirror 13: 0x2caca141 (stale) Mirror 14: 0x2caca141 (stale) Mirror 16: 0x2caca141 (stale) Mirror 17: 0x2caca141 (stale) Mirror 19: 0x2caca141 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1a5): Mirror 7: 0x2caca141 (stale) Mirror 12: 0x2caca141 (stale) Mirror 13: 0x2caca141 (stale) Mirror 14: 0x2caca141 (stale) Mirror 16: 0x2caca141 (stale) Mirror 17: 0x2caca141 (stale) Mirror 19: 0x2caca141 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1a5: skipped 000704 1787830518.615864 mapread 0x011bb4 thru 0x01af7d (0x93ca bytes) 000709 1787830518.640630 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c1a5): Mirror 7: 0x2caca141 (stale) Mirror 12: 0x2caca141 (stale) Mirror 13: 0x2caca141 (stale) Mirror 14: 0x2caca141 (stale) Mirror 16: 0x2caca141 (stale) Mirror 19: 0x2caca141 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1a5): Mirror 7: 0x2caca141 (stale) Mirror 12: 0x2caca141 (stale) Mirror 13: 0x2caca141 (stale) Mirror 14: 0x2caca141 (stale) Mirror 16: 0x2caca141 (stale) Mirror 19: 0x2caca141 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1a5: skipped 000711 1787830518.963210 write 0x02f558 thru 0x039133 (0x9bdc bytes) 000712 1787830518.973533 mapwrite 0x01edc9 thru 0x02c107 (0xd33f bytes) 000713 1787830519.055864 mapread 0x028e6b thru 0x0382e1 (0xf477 bytes) 000714 1787830519.056643 trunc from 0x03c1a5 to 0x037ef0 (0x42b4 bytes) 000715 1787830519.074780 read 0x033218 thru 0x037ef0 (0x4cd9 bytes) 000716 1787830519.079342 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ef1): Mirror 12: 0xa0e291f1 (stale) Mirror 13: 0xa0e291f1 (stale) Mirror 14: 0xa0e291f1 (stale) Mirror 16: 0xa0e291f1 (stale) Mirror 19: 0xa0e291f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ef1): Mirror 12: 0xa0e291f1 (stale) Mirror 13: 0xa0e291f1 (stale) Mirror 14: 0xa0e291f1 (stale) Mirror 16: 0xa0e291f1 (stale) Mirror 19: 0xa0e291f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ef1: skipped 000719 1787830519.523771 read 0x01918c thru 0x01c56f (0x33e4 bytes) 000720 1787830519.532792 mapread 0x00cc57 thru 0x019b43 (0xceed bytes) 000722 1787830519.569242 write 0x026f57 thru 0x027ddb (0xe85 bytes) 000724 1787830519.584696 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ef1): Mirror 13: 0xa0e291f1 (stale) Mirror 14: 0xa0e291f1 (stale) Mirror 16: 0xa0e291f1 (stale) Mirror 19: 0xa0e291f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ef1): Mirror 13: 0xa0e291f1 (stale) Mirror 14: 0xa0e291f1 (stale) Mirror 16: 0xa0e291f1 (stale) Mirror 19: 0xa0e291f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ef1: skipped 000725 1787830520.065518 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37ef1): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ef1): Mirror 13: 0xa0e291f1 (stale) Mirror 14: 0xa0e291f1 (stale) Mirror 16: 0xa0e291f1 (stale) Mirror 19: 0xa0e291f1 (stale) CRC-32 checksum value for chunk [0, 0x37ef1): Mirror 1: 0xb386a824 Mirror 20: 0xb386a824 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ef1: skipped 000727 1787830521.388923 mapwrite 0x012b68 thru 0x01c5bb (0x9a54 bytes) 000728 1787830521.619643 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x37ef1): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ef1): Mirror 13: 0xa0e291f1 (stale) Mirror 14: 0xa0e291f1 (stale) Mirror 16: 0xa0e291f1 (stale) Mirror 19: 0xa0e291f1 (stale) Mirror 20: 0xb386a824 (stale) CRC-32 checksum value for chunk [0, 0x37ef1): Mirror 1: 0x9e339c11 Mirror 21: 0x9e339c11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ef1: skipped 000730 1787830522.464205 trunc from 0x037ef1 to 0x0023e5 (0x35b0b bytes) 000733 1787830522.607338 mapwrite 0x005e92 thru 0x013489 (0xd5f8 bytes) 000734 1787830522.710333 mapwrite 0x021449 thru 0x024517 (0x30cf bytes) 000737 1787830522.765596 read 0x007d1c thru 0x00a353 (0x2638 bytes) 000738 1787830522.766121 mapwrite 0x02bdc1 thru 0x031003 (0x5243 bytes) 000739 1787830522.819648 read 0x00850c thru 0x012d6e (0xa863 bytes) 000740 1787830522.820226 read 0x01391d thru 0x01fc4e (0xc332 bytes) 000741 1787830522.826557 mapwrite 0x02bd7e thru 0x03b3e6 (0xf669 bytes) 000745 1787830522.898634 trunc from 0x03b3e7 to 0x0075d4 (0x33e12 bytes) 000746 1787830522.921494 read 0x003160 thru 0x0075d4 (0x4475 bytes) 000747 1787830522.922161 mapwrite 0x03dac6 thru 0x03f5c3 (0x1afe bytes) 000749 1787830523.009780 trunc from 0x03f5c4 to 0x034bef (0xa9d4 bytes) 000750 1787830523.028303 mapread 0x025ce1 thru 0x02ebba (0x8eda bytes) 000751 1787830523.031624 mapwrite 0x03cb69 thru 0x03ffff (0x3497 bytes) 000752 1787830523.132129 mapwrite 0x00a631 thru 0x0106ed (0x60bd bytes) 000754 1787830523.201544 MIRROR_RESYNC 7 mirrors 000757 1787830524.958011 read 0x0212bd thru 0x030414 (0xf158 bytes) 000760 1787830524.967558 trunc from 0x040000 to 0x01fb29 (0x204d6 bytes) 000761 1787830525.025798 trunc from 0x01fb2a to 0x02ab33 (0xb00a bytes) 000763 1787830525.046029 read 0x0078d1 thru 0x00e33e (0x6a6e bytes) 000764 1787830525.061222 trunc from 0x02ab34 to 0x00ed96 (0x1bd9d bytes) 000766 1787830525.079557 mapwrite 0x031a6d thru 0x03b431 (0x99c5 bytes) 000771 1787830525.159388 read 0x0327bb thru 0x03b431 (0x8c77 bytes) 000772 1787830525.159945 read 0x01100a thru 0x01d15e (0xc155 bytes) 000774 1787830525.172122 read 0x015e4c thru 0x01b141 (0x52f6 bytes) 000779 1787830525.175287 mapwrite 0x0053af thru 0x013f9f (0xebf1 bytes) 000780 1787830525.232594 mapwrite 0x001b76 thru 0x009f5c (0x83e7 bytes) 000781 1787830525.286610 mapread 0x037160 thru 0x03826a (0x110b bytes) 000785 1787830525.289585 mapread 0x029fb1 thru 0x0344b1 (0xa501 bytes) truncating to largest ever: 0x3fcda 000786 1787830525.291520 trunc from 0x03b432 to 0x03fcd9 (0x48a8 bytes) 000787 1787830525.309824 mapwrite 0x020c7a thru 0x0215c7 (0x94e bytes) 000789 1787830525.351936 mapwrite 0x02bce6 thru 0x02d34a (0x1665 bytes) 000790 1787830525.397814 read 0x00b066 thru 0x00cf97 (0x1f32 bytes) 000795 1787830525.402516 write 0x00a805 thru 0x014bf5 (0xa3f1 bytes) 000796 1787830525.404626 mapread 0x01b046 thru 0x01eb3f (0x3afa bytes) 000797 1787830525.405849 read 0x0048b9 thru 0x0135ae (0xecf6 bytes) 000799 1787830525.407859 read 0x01ec15 thru 0x02d4b5 (0xe8a1 bytes) 000800 1787830525.408850 mapwrite 0x03ed2a thru 0x03ffff (0x12d6 bytes) 000802 1787830525.487813 mapwrite 0x00a15c thru 0x00a3eb (0x290 bytes) 000804 1787830525.531433 read 0x03d44d thru 0x03ffff (0x2bb3 bytes) 000806 1787830525.537284 MIRROR_RESYNC 7 mirrors 000807 1787830526.698807 trunc from 0x040000 to 0x01caa4 (0x2355b bytes) 000809 1787830526.791178 write 0x0097d5 thru 0x00b82c (0x2058 bytes) 000810 1787830526.802753 mapwrite 0x01f9e1 thru 0x02059c (0xbbc bytes) 000811 1787830526.914501 MIRROR_RESYNC 7 mirrors 000812 1787830528.230422 mapread 0x00b434 thru 0x018fb6 (0xdb83 bytes) 000814 1787830528.269780 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) 000815 1787830528.702512 MIRROR_EXTEND to 7 mirrors 000816 1787830529.462028 trunc from 0x02059d to 0x0336ea (0x1314e bytes) 000817 1787830529.570578 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x336eb): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x336eb): Mirror 13: 0x3b1f8c4b (stale) Mirror 14: 0x3b1f8c4b (stale) Mirror 16: 0x3b1f8c4b (stale) Mirror 19: 0x3b1f8c4b (stale) Mirror 20: 0x3b1f8c4b (stale) Mirror 21: 0x3b1f8c4b (stale) CRC-32 checksum value for chunk [0, 0x336eb): Mirror 1: 0x3b1f8c4b Mirror 22: 0x3b1f8c4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x336eb: skipped 000818 1787830530.674785 write 0x0277f2 thru 0x036ac9 (0xf2d8 bytes) 000820 1787830530.754771 mapread 0x030b49 thru 0x036ac9 (0x5f81 bytes) 000821 1787830530.755741 read 0x02c55f thru 0x036ac9 (0xa56b bytes) 000822 1787830530.757522 write 0x01c14d thru 0x022b0a (0x69be bytes) 000823 1787830530.776061 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x36aca): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36aca): Mirror 13: 0x14f69c65 (stale) Mirror 14: 0x14f69c65 (stale) Mirror 16: 0x14f69c65 (stale) Mirror 19: 0x14f69c65 (stale) Mirror 20: 0x14f69c65 (stale) Mirror 21: 0x14f69c65 (stale) Mirror 22: 0x14f69c65 (stale) CRC-32 checksum value for chunk [0, 0x36aca): Mirror 1: 0xa0f91f71 Mirror 23: 0xa0f91f71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36aca: skipped 000824 1787830531.795901 MIRROR_RESYNC 9 mirrors 000827 1787830533.038019 trunc from 0x036aca to 0x007b7d (0x2ef4c bytes) 000829 1787830533.114632 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x7b7e): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7b7e): Mirror 13: 0x6452d362 (stale) Mirror 14: 0x6452d362 (stale) Mirror 16: 0x6452d362 (stale) Mirror 19: 0x6452d362 (stale) Mirror 20: 0x6452d362 (stale) Mirror 21: 0x6452d362 (stale) Mirror 22: 0x6452d362 (stale) Mirror 23: 0x6452d362 (stale) CRC-32 checksum value for chunk [0, 0x7b7e): Mirror 1: 0x6452d362 Mirror 24: 0x6452d362 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7b7e: skipped 000830 1787830534.284829 trunc from 0x007b7e to 0x00a986 (0x2e09 bytes) 000831 1787830534.388906 write 0x0044f5 thru 0x0073c6 (0x2ed2 bytes) 000833 1787830534.401048 trunc from 0x00a987 to 0x015bf6 (0xb270 bytes) 000835 1787830534.428855 trunc from 0x015bf7 to 0x0391c9 (0x235d3 bytes) 000837 1787830534.448995 mapwrite 0x0044dc thru 0x0125dc (0xe101 bytes) 000839 1787830534.513416 mapwrite 0x01634b thru 0x022646 (0xc2fc bytes) 000840 1787830534.579851 mapread 0x00e22a thru 0x019cb6 (0xba8d bytes) 000846 1787830534.584392 mapwrite 0x0171d2 thru 0x026990 (0xf7bf bytes) 000847 1787830534.633845 write 0x03a301 thru 0x03ffff (0x5cff bytes) 000849 1787830534.640350 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xee99c6cf (stale) Mirror 14: 0xee99c6cf (stale) Mirror 19: 0xee99c6cf (stale) Mirror 20: 0xee99c6cf (stale) Mirror 21: 0xee99c6cf (stale) Mirror 22: 0xee99c6cf (stale) Mirror 23: 0xee99c6cf (stale) Mirror 24: 0x40bf8ad5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xee99c6cf (stale) Mirror 14: 0xee99c6cf (stale) Mirror 19: 0xee99c6cf (stale) Mirror 20: 0xee99c6cf (stale) Mirror 21: 0xee99c6cf (stale) Mirror 22: 0xee99c6cf (stale) Mirror 23: 0xee99c6cf (stale) Mirror 24: 0x40bf8ad5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000852 1787830535.184954 MIRROR_RESYNC 9 mirrors 000855 1787830536.801837 write 0x00b0f9 thru 0x0188a0 (0xd7a8 bytes) 000857 1787830536.903732 mapwrite 0x0389b6 thru 0x03bf4c (0x3597 bytes) 000858 1787830536.984318 read 0x0099e6 thru 0x0125a1 (0x8bbc bytes) 000860 1787830536.998392 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xa45bf2c1 (stale) Mirror 14: 0xa45bf2c1 (stale) Mirror 19: 0xa45bf2c1 (stale) Mirror 20: 0xa45bf2c1 (stale) Mirror 21: 0xa45bf2c1 (stale) Mirror 22: 0xa45bf2c1 (stale) Mirror 23: 0xa45bf2c1 (stale) Mirror 24: 0xa45bf2c1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x246fc7ae Mirror 25: 0x246fc7ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000864 1787830538.094565 MIRROR_RESYNC 10 mirrors 000865 1787830539.722322 write 0x003607 thru 0x009a66 (0x6460 bytes) 000868 1787830539.845655 mapwrite 0x01c924 thru 0x024e44 (0x8521 bytes) 000869 1787830539.952047 mapread 0x0377ff thru 0x03ab86 (0x3388 bytes) 000870 1787830539.992404 MIRROR_RESYNC 10 mirrors 000871 1787830541.416410 write 0x015a04 thru 0x0200f2 (0xa6ef bytes) 000872 1787830541.528048 mapread 0x0252f3 thru 0x025324 (0x32 bytes) 000873 1787830541.535184 mapread 0x00beba thru 0x019213 (0xd35a bytes) 000876 1787830541.550605 read 0x011f6d thru 0x017fb5 (0x6049 bytes) 000878 1787830541.552446 write 0x00da6f thru 0x0176a8 (0x9c3a bytes) 000879 1787830541.554286 write 0x016eeb thru 0x02053d (0x9653 bytes) 000880 1787830541.556011 mapread 0x00b044 thru 0x01603e (0xaffb bytes) 000881 1787830541.557240 read 0x01e7d2 thru 0x028e50 (0xa67f bytes) 000886 1787830541.582342 trunc from 0x040000 to 0x030e4d (0xf1b2 bytes) 000887 1787830541.601873 write 0x02c476 thru 0x039f73 (0xdafe bytes) 000888 1787830541.603481 write 0x00c346 thru 0x00f065 (0x2d20 bytes) 000889 1787830541.608385 trunc from 0x039f74 to 0x036480 (0x3af3 bytes) 000892 1787830541.637422 mapwrite 0x0346f9 thru 0x037027 (0x292f bytes) 000894 1787830541.726787 write 0x00ee1e thru 0x01d768 (0xe94b bytes) 000895 1787830541.729260 read 0x011f45 thru 0x020fd8 (0xf094 bytes) 000896 1787830541.730083 write 0x00f1b7 thru 0x019d75 (0xabbf bytes) 000897 1787830541.731349 mapwrite 0x025f0c thru 0x02948e (0x3583 bytes) 000898 1787830541.774582 mapread 0x02834b thru 0x02e935 (0x65eb bytes) 000901 1787830541.776524 mapwrite 0x03c1a1 thru 0x03ffff (0x3e5f bytes) 000902 1787830541.840816 mapwrite 0x009902 thru 0x01577a (0xbe79 bytes) 000903 1787830541.889049 read 0x03ea61 thru 0x03ffff (0x159f bytes) 000904 1787830541.889615 mapwrite 0x014de6 thru 0x020a6e (0xbc89 bytes) 000905 1787830541.939341 trunc from 0x040000 to 0x03388f (0xc770 bytes) 000907 1787830541.963876 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33890): Mirror 13: 0xc590c241 (stale) Mirror 14: 0xc590c241 (stale) Mirror 19: 0xc590c241 (stale) Mirror 20: 0xc590c241 (stale) Mirror 21: 0xc590c241 (stale) Mirror 22: 0xc590c241 (stale) Mirror 24: 0xc590c241 (stale) Mirror 25: 0xc590c241 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33890): Mirror 13: 0xc590c241 (stale) Mirror 14: 0xc590c241 (stale) Mirror 19: 0xc590c241 (stale) Mirror 20: 0xc590c241 (stale) Mirror 21: 0xc590c241 (stale) Mirror 22: 0xc590c241 (stale) Mirror 24: 0xc590c241 (stale) Mirror 25: 0xc590c241 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33890: skipped 000909 1787830542.599911 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x33890): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33890): Mirror 13: 0xc590c241 (stale) Mirror 14: 0xc590c241 (stale) Mirror 19: 0xc590c241 (stale) Mirror 20: 0xc590c241 (stale) Mirror 21: 0xc590c241 (stale) Mirror 22: 0xc590c241 (stale) Mirror 24: 0xc590c241 (stale) Mirror 25: 0xc590c241 (stale) CRC-32 checksum value for chunk [0, 0x33890): Mirror 1: 0x9ef25a7f Mirror 26: 0x9ef25a7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33890: skipped 000910 1787830543.817852 trunc from 0x033890 to 0x004e65 (0x2ea2a bytes) 000911 1787830543.908648 read 0x00343f thru 0x004e65 (0x1a27 bytes) 000912 1787830543.915568 mapwrite 0x02bac8 thru 0x038ac7 (0xd000 bytes) 000913 1787830544.013521 mapread 0x032bd7 thru 0x036d11 (0x413b bytes) 000914 1787830544.015330 write 0x026f3d thru 0x032fe4 (0xc0a8 bytes) 000916 1787830544.019035 mapread 0x01c836 thru 0x022af8 (0x62c3 bytes) 000917 1787830544.035394 read 0x01667e thru 0x01fe51 (0x97d4 bytes) 000918 1787830544.036190 mapwrite 0x039c79 thru 0x03ffff (0x6387 bytes) 000919 1787830544.120399 trunc from 0x040000 to 0x02067a (0x1f985 bytes) 000921 1787830544.141534 mapwrite 0x023325 thru 0x033031 (0xfd0d bytes) 000922 1787830544.336445 mapwrite 0x00e4d6 thru 0x017ffa (0x9b25 bytes) 000923 1787830544.410465 read 0x027888 thru 0x033031 (0xb7aa bytes) 000927 1787830544.412549 write 0x03de2e thru 0x03ffff (0x21d2 bytes) 000928 1787830544.415036 write 0x026293 thru 0x034286 (0xdff4 bytes) 000929 1787830544.425138 write 0x03f431 thru 0x03ffff (0xbcf bytes) 000930 1787830544.426129 write 0x00487f thru 0x009ac8 (0x524a bytes) 000931 1787830544.428297 mapread 0x03dc58 thru 0x03ffff (0x23a8 bytes) 000932 1787830544.429372 trunc from 0x040000 to 0x01cc0d (0x233f2 bytes) 000934 1787830544.454643 mapread 0x00efa3 thru 0x0196ad (0xa70b bytes) 000936 1787830544.456664 mapwrite 0x019aa7 thru 0x026b19 (0xd073 bytes) 000937 1787830544.587524 write 0x02862c thru 0x03140c (0x8de1 bytes) 000938 1787830544.588983 read 0x01bb24 thru 0x01de77 (0x2354 bytes) 000941 1787830544.592463 write 0x020f6d thru 0x02753e (0x65d2 bytes) 000942 1787830544.598545 trunc from 0x03140d to 0x0124de (0x1ef2e bytes) 000943 1787830544.616894 mapwrite 0x0383eb thru 0x039cac (0x18c2 bytes) 000945 1787830544.684654 mapwrite 0x02262c thru 0x022cba (0x68f bytes) 000946 1787830544.735510 read 0x02e74f thru 0x030c4f (0x2501 bytes) 000948 1787830544.736629 read 0x003c0e thru 0x01128e (0xd681 bytes) 000950 1787830544.737751 mapread 0x0170f3 thru 0x023dcb (0xccd9 bytes) 000951 1787830544.748309 read 0x019f58 thru 0x01c629 (0x26d2 bytes) 000952 1787830544.748947 read 0x002fb2 thru 0x010d46 (0xdd95 bytes) 000953 1787830544.749964 write 0x03e220 thru 0x03ffff (0x1de0 bytes) 000954 1787830544.751342 write 0x00ac28 thru 0x00f6dc (0x4ab5 bytes) 000957 1787830544.753759 trunc from 0x040000 to 0x0143e8 (0x2bc17 bytes) 000959 1787830544.772489 write 0x016f9b thru 0x01e726 (0x778c bytes) 000961 1787830544.776460 trunc from 0x01e727 to 0x00b935 (0x12df1 bytes) 000963 1787830544.792278 write 0x014042 thru 0x01b042 (0x7001 bytes) 000964 1787830544.802538 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x17e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b043): Mirror 14: 0x3f512e3a (stale) Mirror 19: 0x3f512e3a (stale) Mirror 20: 0x3f512e3a (stale) Mirror 21: 0x3f512e3a (stale) Mirror 22: 0x3f512e3a (stale) Mirror 24: 0x3f512e3a (stale) Mirror 25: 0x3f512e3a (stale) Mirror 26: 0xc13d2b9b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b043): Mirror 14: 0x3f512e3a (stale) Mirror 19: 0x3f512e3a (stale) Mirror 20: 0x3f512e3a (stale) Mirror 21: 0x3f512e3a (stale) Mirror 22: 0x3f512e3a (stale) Mirror 24: 0x3f512e3a (stale) Mirror 25: 0x3f512e3a (stale) Mirror 26: 0xc13d2b9b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b043: skipped 000965 1787830545.379843 write 0x025956 thru 0x030dfe (0xb4a9 bytes) 000966 1787830545.383051 read 0x027226 thru 0x02abf8 (0x39d3 bytes) 000967 1787830545.384355 mapwrite 0x039254 thru 0x03f479 (0x6226 bytes) 000970 1787830545.541266 read 0x025462 thru 0x02dbb7 (0x8756 bytes) 000972 1787830545.543909 write 0x035584 thru 0x036e18 (0x1895 bytes) 000973 1787830545.545992 trunc from 0x03f47a to 0x0260e5 (0x19394 bytes) 000975 1787830545.587822 mapread 0x0192ed thru 0x01ff3e (0x6c52 bytes) 000977 1787830545.604225 mapread 0x0108a6 thru 0x0127b7 (0x1f12 bytes) 000978 1787830545.605929 trunc from 0x0260e6 to 0x005029 (0x210bc bytes) 000979 1787830545.643861 MIRROR_RESYNC 9 mirrors 000980 1787830547.962487 mapread 0x00433c thru 0x005029 (0xcee bytes) 000982 1787830547.979939 write 0x0244ac thru 0x0288be (0x4413 bytes) 000983 1787830548.058412 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x288bf): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x288bf): Mirror 14: 0x4021a287 (stale) Mirror 19: 0x4021a287 (stale) Mirror 20: 0x4021a287 (stale) Mirror 21: 0x4021a287 (stale) Mirror 22: 0x4021a287 (stale) Mirror 24: 0x4021a287 (stale) Mirror 25: 0x4021a287 (stale) Mirror 26: 0x4021a287 (stale) CRC-32 checksum value for chunk [0, 0x288bf): Mirror 1: 0xd126ca0b Mirror 27: 0xd126ca0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x288bf: skipped 000985 1787830549.393398 write 0x017580 thru 0x026743 (0xf1c4 bytes) 000986 1787830549.569352 mapwrite 0x00b2bb thru 0x00b39b (0xe1 bytes) 000987 1787830549.670719 mapread 0x01712f thru 0x0244a8 (0xd37a bytes) 000991 1787830549.679558 read 0x020c5e thru 0x0288be (0x7c61 bytes) 000992 1787830549.702071 mapwrite 0x0175f9 thru 0x01e565 (0x6f6d bytes) 000993 1787830549.790476 mapwrite 0x03b341 thru 0x03c092 (0xd52 bytes) 000995 1787830549.966578 mapread 0x0019bb thru 0x005eef (0x4535 bytes) 000999 1787830549.989581 write 0x0129e3 thru 0x013fd7 (0x15f5 bytes) 001000 1787830550.006520 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3c093): Mirror 14: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c093): Mirror 14: 0x5bed92f4 (stale) Mirror 19: 0x5bed92f4 (stale) Mirror 20: 0x5bed92f4 (stale) Mirror 21: 0x5bed92f4 (stale) Mirror 22: 0x5bed92f4 (stale) Mirror 24: 0x5bed92f4 (stale) Mirror 25: 0x5bed92f4 (stale) Mirror 26: 0x5bed92f4 (stale) Mirror 27: 0xc5f9e4a7 (stale) CRC-32 checksum value for chunk [0, 0x3c093): Mirror 1: 0x5dc4d192 Mirror 28: 0x5dc4d192 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c093: skipped All operations completed A-OK! PASS 70a (128s) == sanity-flr test 71: check mirror extend parent fid ==== 07:35:57 (1787830557) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 07:36:00 (1787830560) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.303428 s, 6.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200551 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 07:36:10 (1787830570) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7290272 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3702080 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8331104 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6314784 .. Extending file size to 7540864 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8630560 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8980640 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9145888 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8943296 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 58677 58678 58681 58682 58683 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) 192.168.202.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:37:29 (1787830649) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9182848 .. Extending file size to 4653408 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6139168 .. Extending file size to 6882720 .. Extending file size to 7037824 .. Extending file size to 7685856 .. Extending file size to 8924640 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60424 60426 60427 60428 60430 done 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) 192.168.202.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:38:52 (1787830732) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129006 s, 16.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.330224 s, 6.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x83bdb843 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:39:13 (1787830753) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109765 s, 19.1 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 07:39:21 (1787830761) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cf:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:39:29 (1787830769) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244886 s, 8.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:39:40 (1787830780) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.240318 s, 8.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:39:50 (1787830790) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.36519 s, 5.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:40:25 (1787830825) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185876 s, 5.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:41:22 (1787830882) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.331345 s, 6.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (67s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:42:29 (1787830949) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg230-server mds-ost sync done. Creating new pool test_204f oleg230-server: Pool lustre.test_204f created Adding targets to pool oleg230-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg230-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141891 s, 14.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg230-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg230-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg230-server: Pool lustre.test_204f destroyed PASS 204f (85s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:43:54 (1787831034) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:44:02 (1787831042) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:44:11 (1787831051) oleg230-server: Pool lustre.test_206 created Adding targets to pool oleg230-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg230-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg230-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg230-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg230-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:44:41 (1787831081) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103805 s, 10.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517982 s, 20.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe95c9c2f Mirror 3: 0xe95c9c2f PASS 207 (16s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:44:57 (1787831097) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:45:01 (1787831101) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:45:04 (1787831104) Starting client: oleg230-client.virtnet: -o user_xattr,flock 192.168.202.130@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0866858 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1e10000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x21a0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x21a0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x21a0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2650000 Read this (4 bytes): BBBB 192.168.202.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg230-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:45:41 (1787831141) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0171946 s, 476 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:45:48 (1787831148) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06763 s, 15.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f4 lmm_fid: [0x200000404:0x1f4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f4 lmm_fid: [0x200000404:0x1f4:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834420 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 07:45:50.000000000 -0400 Modify: 2026-08-27 07:45:50.000000000 -0400 Change: 2026-08-27 07:45:51.000000000 -0400 Birth: 2026-08-27 07:45:50.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:45:56 (1787831156) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:46:01 (1787831161) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.608755 s, 67.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0521763 s, 78.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 07:46 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f6:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 07:46 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:46:10 (1787831170) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:46:17 (1787831177) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1968 sec =========== 07:46:26 (1787831186) === sanity-flr: start cleanup 07:46:28 (1787831188) === === sanity-flr: finish cleanup 07:46:35 (1787831195) ===