-----============= acceptance-small: sanity-flr ============----- Sat Aug 22 06:38:50 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg303-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 06:39:12 (1787395152) === oleg303-client.virtnet: executing check_config_client /mnt/lustre oleg303-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg303-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8aee42e15000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8aee42e15000.idle_timeout=debug disable quota as required oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 06:39:53 (1787395193) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7608] == sanity-flr test 0a: lfs mirror create with -N option == 06:39:56 (1787395196) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 06:40:16 (1787395216) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 06:40:20 (1787395220) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 06:40:24 (1787395224) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 06:40:49 (1787395249) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 06:40:51 (1787395251) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 06:40:54 (1787395254) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8aee42e15000.stats=clear osc.lustre-OST0001-osc-ffff8aee42e15000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 06:41:00 (1787395260) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 06:41:11 (1787395271) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 06:41:20 (1787395280) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 06:41:23 (1787395283) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 06:41:30 (1787395290) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 06:41:40 (1787395300) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 06:41:47 (1787395307) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 06:41:51 (1787395311) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 06:41:58 (1787395318) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 06:42:08 (1787395328) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 06:42:14 (1787395334) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.96291 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218667 s, 4.8 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 06:42:26 (1787395346) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221583 s, 47.3 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 06:42:33 (1787395353) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 06:42:44 (1787395364) Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8aee42e15000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8aee42e15000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg303-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8aee42e15000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8aee42e15000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (137s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 06:45:00 (1787395500) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.264538 s, 19.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8aee42e15000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8aee42e15000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 06:45:41 (1787395541) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8aee42e15000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8aee42e15000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8aee42e15000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8aee42e15000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 06:46:57 (1787395617) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.79164 s, 10.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 22 06:47 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.059s user 0m0.001s sys 0m0.157s Stopping /mnt/lustre-ost1 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8aee42e15000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8aee42e15000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.064s user 0m0.001s sys 0m0.187s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg303-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg303-server oleg303-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8aee42e15000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8aee42e15000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.079s user 0m0.002s sys 0m0.182s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg303-server: oleg303-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg303-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg303-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8aee42e15000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8aee42e15000.ost_server_uuid in FULL state after 0 sec PASS 33b (83s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 06:48:19 (1787395699) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 06:48:23 (1787395703) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 06:48:27 (1787395707) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 06:48:30 (1787395710) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 06:48:38 (1787395718) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.587279 s, 21.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 06:48:51 (1787395731) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364034 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025563 s, 41.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 06:48:58 (1787395738) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 06:49:06 (1787395746) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00124666 s, 821 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000793563 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00451169 s, 227 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00289871 s, 353 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 06:49:16 (1787395756) create test files with size 15757523 .. 3a2c526c648a0c46296e788faf689a13 - 1c3ee696e719192b2aab5a20bd0614dd - 0951322eade0a1d2881cf3f64f202ba9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 06:49:40 (1787395780) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 06:49:56 (1787395796) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 06:50:03 (1787395803) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141737 s, 7.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034312 s, 30.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 06:50:13 (1787395813) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192771 s, 10.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.473848 s, 8.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.316319 s, 6.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (21s) == sanity-flr test 42: lfs mirror verify ================= 06:50:35 (1787395835) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 06:50:39 (1787395839) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 06:50:43 (1787395843) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409224 s, 25.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 06:50:51 (1787395851) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 06:50:55 (1787395855) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 06:51:04 (1787395864) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.219725 s, 47.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 06:51:17 (1787395877) llite.lustre-ffff8aee433f6800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.31284 s, 33.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg303-client.virtnet /mnt/lustre (opts:) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8aee50acb000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 06:51:31 (1787395891) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (34s) == sanity-flr test 46: Verify setstripe --copy option ==== 06:52:05 (1787395925) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 06:52:41 (1787395961) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 06:52:45 (1787395965) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.201273 s, 15.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.182964 s, 17.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x37b08561 (nosync,stale) Mirror 2: 0x37b08561 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 54ac0b6a3129caa6b7cead6efc747be0 - mirror 1 checksum: 78757f4dd2aad2f292990a9e85a2601a - mirror 2 checksum: 54ac0b6a3129caa6b7cead6efc747be0 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T06:52:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T06:52:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 54ac0b6a3129caa6b7cead6efc747be0 - mirror 1 checksum: 78757f4dd2aad2f292990a9e85a2601a - mirror 2 checksum: 78757f4dd2aad2f292990a9e85a2601a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T06:52:48-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T06:52:48-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 06:52:58 (1787395978) Failing mds1 on oleg303-server Stopping /mnt/lustre-mds1 (opts:) on oleg303-server 06:53:04 (1787395984) shut down facet: mds1 facet_host: oleg303-server facet_failover_host: oleg303-server Failover mds1 to oleg303-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 06:53:27 (1787396007) targets are mounted 06:53:27 (1787396007) facet_failover done oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555847 s, 18.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 06:53:45 (1787396025) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 06:53:48 (1787396028) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 06:53:51 (1787396031) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00731617 s, 2.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 06:53:59 (1787396039) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 06:54:02 (1787396042) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 06:54:05 (1787396045) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 06:54:29 (1787396069) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 06:54:56 (1787396096) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 06:55:20 (1787396120) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 06:55:43 (1787396143) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 06:55:50 (1787396150) mirror create pid 53767 mirror split pid 53768 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53767 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53768 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 06:56:56 (1787396216) Seed set to 6217 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero size read 000014 1787396218.245482 write 0x02ad3e thru 0x02f833 (0x4af6 bytes) 000015 1787396218.251483 write 0x03858d thru 0x03ffff (0x7a73 bytes) 000017 1787396218.254491 write 0x00c44b thru 0x0111ce (0x4d84 bytes) 000019 1787396218.284444 write 0x029cc5 thru 0x03147f (0x77bb bytes) 000021 1787396218.299223 mapread 0x02869b thru 0x035cc9 (0xd62f bytes) 000023 1787396218.329024 mapread 0x02d482 thru 0x03a273 (0xcdf2 bytes) 000024 1787396218.330827 read 0x03ed91 thru 0x03ffff (0x126f bytes) 000028 1787396218.338029 read 0x00e959 thru 0x0118eb (0x2f93 bytes) 000029 1787396218.338837 mapwrite 0x03529f thru 0x03d4cb (0x822d bytes) 000031 1787396218.411919 write 0x006761 thru 0x012717 (0xbfb7 bytes) 000033 1787396218.426076 write 0x0037ef thru 0x0045b4 (0xdc6 bytes) 000034 1787396218.427043 mapread 0x001c30 thru 0x0090a3 (0x7474 bytes) 000036 1787396218.441067 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x3993b 000037 1787396219.181450 trunc from 0x040000 to 0x03993a (0x66c5 bytes) 000038 1787396219.320032 trunc from 0x03993b to 0x015339 (0x24601 bytes) 000040 1787396219.353543 mapwrite 0x036f7d thru 0x03c372 (0x53f6 bytes) 000041 1787396219.458525 mapread 0x012e91 thru 0x01a0a2 (0x7212 bytes) 000042 1787396219.469865 mapread 0x03be6b thru 0x03c372 (0x508 bytes) 000043 1787396219.471081 trunc from 0x03c373 to 0x018588 (0x23dea bytes) 000046 1787396219.499970 read 0x016af9 thru 0x018588 (0x1a90 bytes) 000049 1787396219.504893 read 0x007257 thru 0x00bd43 (0x4aed bytes) truncating to largest ever: 0x3b180 000053 1787396219.527182 trunc from 0x018589 to 0x03b17f (0x22bf7 bytes) 000056 1787396219.592066 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b180): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b180): Mirror 2: 0x53b612fd (stale) CRC-32 checksum value for chunk [0, 0x3b180): Mirror 1: 0x48cd9414 Mirror 3: 0x48cd9414 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b180: skipped 000057 1787396220.344844 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b180): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b180): Mirror 2: 0x53b612fd (stale) CRC-32 checksum value for chunk [0, 0x3b180): Mirror 1: 0x48cd9414 Mirror 3: 0x48cd9414 Mirror 4: 0x48cd9414 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b180: skipped 000060 1787396221.189741 MIRROR_RESYNC 4 mirrors 000061 1787396221.933193 mapread 0x0188e5 thru 0x028772 (0xfe8e bytes) 000062 1787396221.973398 mapread 0x023818 thru 0x02409c (0x885 bytes) 000063 1787396221.974100 mapread 0x0340b9 thru 0x03b17f (0x70c7 bytes) 000064 1787396221.996726 write 0x033490 thru 0x033c9f (0x810 bytes) 000066 1787396222.117838 trunc from 0x03b180 to 0x02ab8d (0x105f2 bytes) 000067 1787396222.156079 mapread 0x01b616 thru 0x02694d (0xb338 bytes) 000068 1787396222.177792 MIRROR_RESYNC 4 mirrors 000069 1787396223.023354 mapwrite 0x01abe2 thru 0x02000d (0x542c bytes) 000071 1787396223.265244 write 0x017fbf thru 0x01e005 (0x6047 bytes) 000073 1787396223.270614 write 0x0296e8 thru 0x0384b8 (0xedd1 bytes) 000076 1787396223.286752 read 0x0294ec thru 0x02c0b0 (0x2bc5 bytes) 000077 1787396223.291060 write 0x03e8d0 thru 0x03eca5 (0x3d6 bytes) 000081 1787396223.296800 trunc from 0x03eca6 to 0x03a5d3 (0x46d2 bytes) 000082 1787396223.360611 read 0x00d759 thru 0x013908 (0x61b0 bytes) truncating to largest ever: 0x3c811 000084 1787396223.374530 trunc from 0x03a5d4 to 0x03c810 (0x223d bytes) 000085 1787396223.435769 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c811): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c811): Mirror 2: 0x17c921a3 (stale) Mirror 3: 0x17c921a3 (stale) Mirror 4: 0x17c921a3 (stale) CRC-32 checksum value for chunk [0, 0x3c811): Mirror 1: 0x2f1423c2 Mirror 5: 0x2f1423c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c811: skipped 000086 1787396224.398516 write 0x01333b thru 0x014c1a (0x18e0 bytes) 000087 1787396224.516441 mapread 0x02b1f2 thru 0x03a0ef (0xeefe bytes) 000090 1787396224.577303 read 0x013e37 thru 0x01bc45 (0x7e0f bytes) 000092 1787396224.618762 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c811): Mirror 3: 0x17c921a3 (stale) Mirror 4: 0x17c921a3 (stale) Mirror 5: 0x2f1423c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c811): Mirror 3: 0x17c921a3 (stale) Mirror 4: 0x17c921a3 (stale) Mirror 5: 0x2f1423c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c811: skipped 000093 1787396225.064764 mapwrite 0x0163af thru 0x01c3b3 (0x6005 bytes) 000094 1787396225.184603 write 0x024dde thru 0x02b31c (0x653f bytes) 000095 1787396225.205590 mapwrite 0x00cfdb thru 0x018747 (0xb76d bytes) 000096 1787396225.278849 MIRROR_RESYNC 4 mirrors 000097 1787396226.129573 trunc from 0x03c811 to 0x032ca7 (0x9b69 bytes) 000098 1787396226.259032 write 0x024225 thru 0x02bff6 (0x7dd2 bytes) 000100 1787396226.276883 read 0x012e20 thru 0x01cd72 (0x9f53 bytes) 000101 1787396226.301818 trunc from 0x032ca8 to 0x028938 (0xa36f bytes) 000104 1787396226.342493 read 0x006183 thru 0x00f0c3 (0x8f41 bytes) 000105 1787396226.359820 write 0x031b16 thru 0x035c4e (0x4139 bytes) 000106 1787396226.361066 mapwrite 0x025535 thru 0x030ae8 (0xb5b4 bytes) 000108 1787396226.440418 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x35c4f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c4f): Mirror 3: 0x3ba0632 (stale) Mirror 4: 0x3ba0632 (stale) Mirror 5: 0x3ba0632 (stale) CRC-32 checksum value for chunk [0, 0x35c4f): Mirror 1: 0x602a90df Mirror 6: 0x602a90df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c4f: skipped 000109 1787396227.157204 trunc from 0x035c4f to 0x03843b (0x27ed bytes) 000110 1787396227.331538 mapread 0x01ed92 thru 0x026d25 (0x7f94 bytes) 000111 1787396227.359766 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3843c): Mirror 4: 0xa14c52ae (stale) Mirror 5: 0xa14c52ae (stale) Mirror 6: 0x2bef3e08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3843c): Mirror 4: 0xa14c52ae (stale) Mirror 5: 0xa14c52ae (stale) Mirror 6: 0x2bef3e08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3843c: skipped 000112 1787396227.819491 read 0x01bccd thru 0x024841 (0x8b75 bytes) 000113 1787396227.830579 mapwrite 0x02a84a thru 0x0322a7 (0x7a5e bytes) 000114 1787396227.892694 mapread 0x027d1b thru 0x035f31 (0xe217 bytes) 000115 1787396227.905716 read 0x00c8c8 thru 0x00f6d1 (0x2e0a bytes) 000117 1787396227.919466 trunc from 0x03843c to 0x020e7d (0x175be bytes) 000118 1787396227.945556 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x20e7e): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20e7e): Mirror 4: 0x97a908f0 (stale) Mirror 5: 0x97a908f0 (stale) Mirror 6: 0x97a908f0 (stale) CRC-32 checksum value for chunk [0, 0x20e7e): Mirror 1: 0x97a908f0 Mirror 7: 0x97a908f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20e7e: skipped 000119 1787396228.986462 trunc from 0x020e7e to 0x03b947 (0x1aaca bytes) 000121 1787396229.097699 read 0x01de8e thru 0x0286a3 (0xa816 bytes) 000122 1787396229.113023 read 0x039974 thru 0x03b183 (0x1810 bytes) 000124 1787396229.136616 mapread 0x02eb8e thru 0x030be7 (0x205a bytes) 000125 1787396229.138064 mapread 0x023f91 thru 0x026899 (0x2909 bytes) 000126 1787396229.140485 mapread 0x039843 thru 0x03b947 (0x2105 bytes) 000132 1787396229.147256 trunc from 0x03b948 to 0x00fda4 (0x2bba3 bytes) 000134 1787396229.176468 MIRROR_RESYNC 5 mirrors 000135 1787396230.417860 write 0x026f7e thru 0x02c58e (0x5611 bytes) 000136 1787396230.535133 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c58f): Mirror 4: 0x64ff5efd (stale) Mirror 5: 0x64ff5efd (stale) Mirror 6: 0x64ff5efd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c58f): Mirror 4: 0x64ff5efd (stale) Mirror 5: 0x64ff5efd (stale) Mirror 6: 0x64ff5efd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c58f: skipped 000137 1787396230.944892 mapwrite 0x014083 thru 0x019c7f (0x5bfd bytes) 000138 1787396231.066242 mapread 0x009b0d thru 0x0156fc (0xbbf0 bytes) 000139 1787396231.099940 MIRROR_RESYNC 4 mirrors 000140 1787396232.138978 mapread 0x029b34 thru 0x02b23a (0x1707 bytes) 000141 1787396232.158651 mapwrite 0x031c5b thru 0x03d606 (0xb9ac bytes) 000142 1787396232.468034 mapread 0x037a77 thru 0x03ca8f (0x5019 bytes) 000144 1787396232.471304 mapread 0x022298 thru 0x029afe (0x7867 bytes) 000147 1787396232.483543 read 0x039d2a thru 0x03d606 (0x38dd bytes) 000150 1787396232.487716 mapwrite 0x0205be thru 0x025c00 (0x5643 bytes) 000151 1787396232.530763 write 0x01fe54 thru 0x02901d (0x91ca bytes) 000153 1787396232.532786 write 0x007a59 thru 0x00b6f7 (0x3c9f bytes) 000154 1787396232.555315 MIRROR_RESYNC 4 mirrors 000155 1787396233.679052 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) 000156 1787396234.149897 MIRROR_EXTEND to 4 mirrors 000157 1787396234.882513 trunc from 0x03d607 to 0x03b404 (0x2202 bytes) 000158 1787396234.967843 mapread 0x01990c thru 0x021640 (0x7d35 bytes) 000159 1787396234.992086 write 0x02a43e thru 0x034409 (0x9fcc bytes) 000160 1787396235.004429 mapwrite 0x00b6d4 thru 0x012147 (0x6a74 bytes) 000162 1787396235.080507 write 0x0041a8 thru 0x00c980 (0x87d9 bytes) 000164 1787396235.087198 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3b405): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b405): Mirror 5: 0xfc0ddfb2 (stale) Mirror 6: 0xfc0ddfb2 (stale) Mirror 7: 0xfc0ddfb2 (stale) CRC-32 checksum value for chunk [0, 0x3b405): Mirror 1: 0xf1b73c51 Mirror 8: 0xf1b73c51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b405: skipped 000166 1787396236.092007 write 0x0223e8 thru 0x02a17c (0x7d95 bytes) 000167 1787396236.236217 mapread 0x0221b9 thru 0x0315cf (0xf417 bytes) 000169 1787396236.271904 mapread 0x0276f4 thru 0x02a9a7 (0x32b4 bytes) 000171 1787396236.275048 mapread 0x03af88 thru 0x03b190 (0x209 bytes) 000172 1787396236.286529 trunc from 0x03b405 to 0x01b53c (0x1fec8 bytes) 000176 1787396236.365023 mapread 0x01a8a9 thru 0x01b53c (0xc94 bytes) 000177 1787396236.383640 read 0x012639 thru 0x017ba5 (0x556d bytes) 000179 1787396236.403070 read 0x01b4bb thru 0x01b53c (0x82 bytes) 000181 1787396236.404451 write 0x008a7e thru 0x00bb02 (0x3085 bytes) 000184 1787396236.415752 trunc from 0x01b53d to 0x01b79b (0x25f bytes) 000185 1787396236.437349 trunc from 0x01b79c to 0x03baea (0x2034f bytes) 000189 1787396236.482652 trunc from 0x03baeb to 0x0108bf (0x2b22b bytes) 000190 1787396236.503464 trunc from 0x0108c0 to 0x027792 (0x16ed3 bytes) 000191 1787396236.529303 write 0x00a92f thru 0x01a504 (0xfbd6 bytes) 000192 1787396236.547382 write 0x011516 thru 0x021450 (0xff3b bytes) 000193 1787396236.550540 trunc from 0x027793 to 0x014490 (0x13302 bytes) 000194 1787396236.587208 mapread 0x006f5c thru 0x00ea7a (0x7b1f bytes) 000196 1787396236.590625 read 0x00527d thru 0x00695a (0x16de bytes) 000197 1787396236.591982 mapread 0x00cc28 thru 0x014490 (0x7869 bytes) 000198 1787396236.594044 trunc from 0x014491 to 0x034e6d (0x209dd bytes) 000199 1787396236.615332 write 0x01b654 thru 0x026f15 (0xb8c2 bytes) 000200 1787396236.632239 MIRROR_RESYNC 5 mirrors 000201 1787396237.715645 write 0x01ca3b thru 0x020b00 (0x40c6 bytes) 000204 1787396237.788110 mapread 0x02e828 thru 0x034e6d (0x6646 bytes) 000205 1787396237.818144 write 0x01d5b7 thru 0x021dbe (0x4808 bytes) 000206 1787396237.824365 trunc from 0x034e6e to 0x00cd75 (0x280f8 bytes) 000207 1787396237.848869 mapread 0x00a11d thru 0x00cd75 (0x2c59 bytes) 000208 1787396237.861625 read 0x000f17 thru 0x0035cd (0x26b7 bytes) 000209 1787396237.868493 write 0x01b527 thru 0x026aae (0xb588 bytes) 000211 1787396237.871213 trunc from 0x026aaf to 0x02ff87 (0x94d9 bytes) 000212 1787396237.888762 write 0x02ae41 thru 0x02bd7b (0xf3b bytes) 000214 1787396237.904600 mapread 0x023191 thru 0x02c8ff (0x976f bytes) 000215 1787396237.922039 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ff88): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff88): Mirror 5: 0xc2114a49 (stale) Mirror 6: 0xc2114a49 (stale) Mirror 7: 0xc2114a49 (stale) Mirror 8: 0xc2114a49 (stale) CRC-32 checksum value for chunk [0, 0x2ff88): Mirror 1: 0xff64356d Mirror 9: 0xff64356d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff88: skipped 000216 1787396238.862168 read 0x000612 thru 0x00601f (0x5a0e bytes) 000218 1787396238.909946 read 0x011afc thru 0x021374 (0xf879 bytes) 000219 1787396239.000143 mapread 0x00a1ae thru 0x00aedd (0xd30 bytes) 000220 1787396239.029376 trunc from 0x02ff88 to 0x01b8be (0x146c9 bytes) 000224 1787396239.170668 mapwrite 0x030800 thru 0x031d15 (0x1516 bytes) 000228 1787396239.331926 read 0x02888c thru 0x031d15 (0x948a bytes) 000232 1787396239.348021 mapwrite 0x015a6c thru 0x015d88 (0x31d bytes) 000234 1787396239.439489 trunc from 0x031d16 to 0x02dd19 (0x3ffc bytes) 000236 1787396239.469279 write 0x03ec8d thru 0x03ffff (0x1373 bytes) 000238 1787396239.472034 mapwrite 0x01d555 thru 0x023f80 (0x6a2c bytes) 000239 1787396239.520698 mapwrite 0x016df1 thru 0x02139d (0xa5ad bytes) 000240 1787396239.572335 read 0x023be2 thru 0x02d77f (0x9b9e bytes) 000242 1787396239.576945 trunc from 0x040000 to 0x00b56d (0x34a92 bytes) truncating to largest ever: 0x3d3ee 000243 1787396239.611496 trunc from 0x00b56e to 0x03d3ed (0x31e80 bytes) 000245 1787396239.646556 trunc from 0x03d3ee to 0x02c58b (0x10e62 bytes) 000247 1787396239.685509 trunc from 0x02c58c to 0x0341d4 (0x7c49 bytes) 000248 1787396239.713080 mapread 0x026045 thru 0x02cd24 (0x6ce0 bytes) 000249 1787396239.727678 trunc from 0x0341d5 to 0x02a7ed (0x99e7 bytes) 000251 1787396239.746911 read 0x004022 thru 0x012b7d (0xeb5c bytes) 000252 1787396239.757629 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a7ee): Mirror 5: 0xcad38d94 (stale) Mirror 6: 0xcad38d94 (stale) Mirror 7: 0xcad38d94 (stale) Mirror 9: 0x9c3e0d2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a7ee): Mirror 5: 0xcad38d94 (stale) Mirror 6: 0xcad38d94 (stale) Mirror 7: 0xcad38d94 (stale) Mirror 9: 0x9c3e0d2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a7ee: skipped 000254 1787396240.333142 write 0x03d7bd thru 0x03ffff (0x2843 bytes) 000255 1787396240.345689 MIRROR_RESYNC 5 mirrors 000256 1787396241.518597 trunc from 0x040000 to 0x012ebd (0x2d142 bytes) 000258 1787396241.704467 mapwrite 0x0304a7 thru 0x03644c (0x5fa6 bytes) 000261 1787396241.990209 mapwrite 0x029fd9 thru 0x03598d (0xb9b5 bytes) 000262 1787396242.130699 trunc from 0x03644d to 0x00fd62 (0x266ea bytes) 000263 1787396242.169132 read 0x00456c thru 0x00e833 (0xa2c8 bytes) 000264 1787396242.200561 read 0x00cdb1 thru 0x00de8c (0x10dc bytes) 000265 1787396242.201626 read 0x001fc6 thru 0x00fd62 (0xdd9d bytes) 000266 1787396242.222119 mapwrite 0x03d730 thru 0x03ffff (0x28d0 bytes) 000267 1787396242.360934 mapread 0x01d255 thru 0x021757 (0x4503 bytes) 000268 1787396242.365590 mapwrite 0x03d543 thru 0x03ffff (0x2abd bytes) 000271 1787396242.428785 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x97e405cb (stale) Mirror 6: 0x97e405cb (stale) Mirror 7: 0x97e405cb (stale) Mirror 9: 0x97e405cb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x98263a8a Mirror 10: 0x98263a8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000276 1787396243.566440 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x97e405cb (stale) Mirror 6: 0x97e405cb (stale) Mirror 7: 0x97e405cb (stale) Mirror 9: 0x97e405cb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x98263a8a Mirror 10: 0x98263a8a Mirror 11: 0x98263a8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000277 1787396244.597501 trunc from 0x040000 to 0x022fcf (0x1d030 bytes) 000280 1787396244.739828 write 0x016b4c thru 0x01c1ee (0x56a3 bytes) 000282 1787396244.760679 write 0x02f7f5 thru 0x039725 (0x9f31 bytes) 000283 1787396244.768981 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x39726): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39726): Mirror 5: 0xe228855f (stale) Mirror 6: 0xe228855f (stale) Mirror 7: 0xe228855f (stale) Mirror 9: 0xe228855f (stale) Mirror 10: 0xe228855f (stale) Mirror 11: 0xe228855f (stale) CRC-32 checksum value for chunk [0, 0x39726): Mirror 1: 0xacece0d3 Mirror 12: 0xacece0d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39726: skipped 000284 1787396245.875983 trunc from 0x039726 to 0x024ee9 (0x1483c bytes) 000286 1787396245.989801 read 0x01816c thru 0x022f63 (0xadf8 bytes) 000289 1787396246.007253 trunc from 0x024eea to 0x0113a3 (0x13b46 bytes) 000292 1787396246.030452 read 0x00301f thru 0x00f96c (0xc94e bytes) 000295 1787396246.048649 trunc from 0x0113a4 to 0x01cd2e (0xb98b bytes) 000297 1787396246.105638 MIRROR_RESYNC 8 mirrors 000299 1787396247.484423 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) 000300 1787396247.860317 mapwrite 0x023888 thru 0x02805f (0x47d8 bytes) 000305 1787396248.033655 mapwrite 0x01d63d thru 0x02d534 (0xfef8 bytes) 000309 1787396248.148615 trunc from 0x02d535 to 0x030b27 (0x35f3 bytes) 000310 1787396248.178145 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30b28): Mirror 5: 0x2a457cf6 (stale) Mirror 7: 0x2a457cf6 (stale) Mirror 9: 0x2a457cf6 (stale) Mirror 11: 0x2a457cf6 (stale) Mirror 12: 0x2a457cf6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b28): Mirror 5: 0x2a457cf6 (stale) Mirror 7: 0x2a457cf6 (stale) Mirror 9: 0x2a457cf6 (stale) Mirror 11: 0x2a457cf6 (stale) Mirror 12: 0x2a457cf6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b28: skipped 000311 1787396248.674392 write 0x022d4d thru 0x02d974 (0xac28 bytes) 000312 1787396248.699291 write 0x03b2ad thru 0x03ffff (0x4d53 bytes) 000313 1787396248.706856 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x4c7b42aa (stale) Mirror 7: 0x4c7b42aa (stale) Mirror 11: 0x4c7b42aa (stale) Mirror 12: 0x4c7b42aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x4c7b42aa (stale) Mirror 7: 0x4c7b42aa (stale) Mirror 11: 0x4c7b42aa (stale) Mirror 12: 0x4c7b42aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000314 1787396249.343971 read 0x00e568 thru 0x013433 (0x4ecc bytes) 000315 1787396249.362821 mapread 0x013e9d thru 0x018761 (0x48c5 bytes) 000316 1787396249.401188 mapread 0x014b36 thru 0x016b7b (0x2046 bytes) 000319 1787396249.407474 mapwrite 0x010af7 thru 0x011392 (0x89c bytes) 000320 1787396249.460439 mapread 0x012cbe thru 0x019524 (0x6867 bytes) 000321 1787396249.462874 mapread 0x02ad6f thru 0x0356b8 (0xa94a bytes) 000322 1787396249.483049 read 0x01f417 thru 0x02a651 (0xb23b bytes) 000323 1787396249.496808 write 0x00fdf5 thru 0x0107e9 (0x9f5 bytes) 000324 1787396249.517468 MIRROR_RESYNC 5 mirrors 000329 1787396251.162812 mapwrite 0x02f99a thru 0x031421 (0x1a88 bytes) 000331 1787396251.399127 mapwrite 0x034eaf thru 0x03a74d (0x589f bytes) 000332 1787396251.573796 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x74ea2f69 (stale) Mirror 7: 0x74ea2f69 (stale) Mirror 11: 0x74ea2f69 (stale) Mirror 12: 0x74ea2f69 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xba39d36a Mirror 13: 0xba39d36a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000334 1787396252.654032 mapread 0x01e684 thru 0x02ad31 (0xc6ae bytes) 000335 1787396252.742616 mapread 0x0272f0 thru 0x031f05 (0xac16 bytes) 000337 1787396252.754274 mapread 0x028abf thru 0x032bc9 (0xa10b bytes) 000338 1787396252.771581 mapread 0x0291b9 thru 0x03737d (0xe1c5 bytes) 000340 1787396252.792387 read 0x01cf2c thru 0x025fa3 (0x9078 bytes) 000341 1787396252.799403 read 0x0020d2 thru 0x009bdc (0x7b0b bytes) 000342 1787396252.811981 trunc from 0x040000 to 0x01f17f (0x20e80 bytes) 000343 1787396252.997579 write 0x022f7a thru 0x02b5aa (0x8631 bytes) 000344 1787396253.002417 read 0x0095f0 thru 0x0172e6 (0xdcf7 bytes) 000345 1787396253.018099 mapwrite 0x03dff7 thru 0x03ffff (0x2009 bytes) 000346 1787396253.138009 MIRROR_RESYNC 6 mirrors 000347 1787396254.863023 trunc from 0x040000 to 0x01af43 (0x250bc bytes) 000350 1787396254.989768 mapread 0x00cd64 thru 0x01643f (0x96dc bytes) 000353 1787396255.024367 read 0x01a28e thru 0x01af43 (0xcb6 bytes) 000354 1787396255.032825 mapwrite 0x00eb5f thru 0x0168f2 (0x7d94 bytes) 000357 1787396255.146818 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1af44): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1af44): Mirror 5: 0x533dc711 (stale) Mirror 7: 0x533dc711 (stale) Mirror 11: 0x533dc711 (stale) Mirror 12: 0x533dc711 (stale) Mirror 13: 0x533dc711 (stale) CRC-32 checksum value for chunk [0, 0x1af44): Mirror 1: 0xe8861773 Mirror 14: 0xe8861773 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1af44: skipped 000358 1787396256.587006 read 0x013514 thru 0x018c12 (0x56ff bytes) 000360 1787396256.617566 mapread 0x013480 thru 0x01924c (0x5dcd bytes) 000362 1787396256.631066 write 0x008608 thru 0x00b309 (0x2d02 bytes) 000363 1787396256.826581 write 0x01d888 thru 0x01e1b6 (0x92f bytes) 000365 1787396256.833975 read 0x004659 thru 0x00b6b8 (0x7060 bytes) 000366 1787396256.847331 read 0x000d67 thru 0x00de92 (0xd12c bytes) 000367 1787396256.848134 mapread 0x01b489 thru 0x01cc10 (0x1788 bytes) 000368 1787396256.852272 write 0x010225 thru 0x020094 (0xfe70 bytes) 000370 1787396256.859654 MIRROR_RESYNC 7 mirrors 000371 1787396259.972040 write 0x037d74 thru 0x03ffff (0x828c bytes) 000373 1787396260.142145 MIRROR_RESYNC 7 mirrors 000374 1787396262.354672 write 0x01ec7a thru 0x022111 (0x3498 bytes) 000376 1787396262.541435 trunc from 0x040000 to 0x031611 (0xe9ee bytes) 000381 1787396262.581871 mapwrite 0x039b0b thru 0x03ffff (0x64f5 bytes) 000382 1787396262.749256 write 0x00bfb7 thru 0x00c167 (0x1b1 bytes) 000383 1787396262.759718 write 0x00b8d5 thru 0x00cd74 (0x14a0 bytes) 000384 1787396262.760816 mapwrite 0x023991 thru 0x0268c6 (0x2f36 bytes) 000385 1787396262.853023 trunc from 0x040000 to 0x025bd9 (0x1a426 bytes) 000387 1787396262.888213 mapread 0x0145f6 thru 0x0244f4 (0xfeff bytes) 000388 1787396262.908714 MIRROR_RESYNC 7 mirrors 000389 1787396264.252816 read 0x022e2d thru 0x025bd9 (0x2dad bytes) 000391 1787396264.267741 read 0x0165c3 thru 0x025a25 (0xf463 bytes) 000392 1787396264.296575 trunc from 0x025bda to 0x01afe9 (0xabf0 bytes) 000393 1787396264.447899 MIRROR_RESYNC 7 mirrors 000394 1787396266.100036 write 0x00a07d thru 0x014a9a (0xaa1e bytes) 000395 1787396266.245231 write 0x0386c9 thru 0x03bdfc (0x3734 bytes) 000398 1787396266.251963 read 0x010eb5 thru 0x01a6c8 (0x9814 bytes) 000400 1787396266.268855 trunc from 0x03bdfd to 0x0164ec (0x25910 bytes) 000401 1787396266.301230 mapwrite 0x0372c4 thru 0x038212 (0xf4f bytes) 000403 1787396266.415763 mapwrite 0x0251a0 thru 0x02ba51 (0x68b2 bytes) 000404 1787396266.511303 read 0x004a06 thru 0x01227d (0xd878 bytes) 000406 1787396266.537541 read 0x0062b9 thru 0x00a013 (0x3d5b bytes) 000407 1787396266.539016 mapwrite 0x02247b thru 0x028d32 (0x68b8 bytes) 000408 1787396266.622390 trunc from 0x038213 to 0x02194a (0x168c8 bytes) 000410 1787396266.653555 mapread 0x01125f thru 0x01ccbe (0xba60 bytes) 000412 1787396266.664222 read 0x01d7d0 thru 0x02194a (0x417b bytes) 000415 1787396266.677543 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2194b): Mirror 5: 0xf1967e11 (stale) Mirror 7: 0xf1967e11 (stale) Mirror 11: 0xf1967e11 (stale) Mirror 12: 0xf1967e11 (stale) Mirror 13: 0xf1967e11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2194b): Mirror 5: 0xf1967e11 (stale) Mirror 7: 0xf1967e11 (stale) Mirror 11: 0xf1967e11 (stale) Mirror 12: 0xf1967e11 (stale) Mirror 13: 0xf1967e11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2194b: skipped 000416 1787396267.154666 read 0x020adc thru 0x02194a (0xe6f bytes) 000422 1787396267.178293 mapread 0x01e18f thru 0x02194a (0x37bc bytes) 000423 1787396267.212966 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2194b): Mirror 5: 0xf1967e11 (stale) Mirror 7: 0xf1967e11 (stale) Mirror 11: 0xf1967e11 (stale) Mirror 13: 0xf1967e11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2194b): Mirror 5: 0xf1967e11 (stale) Mirror 7: 0xf1967e11 (stale) Mirror 11: 0xf1967e11 (stale) Mirror 13: 0xf1967e11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2194b: skipped 000424 1787396267.743257 MIRROR_RESYNC 5 mirrors 000425 1787396268.977739 read 0x01f045 thru 0x02194a (0x2906 bytes) 000426 1787396268.994372 MIRROR_RESYNC 5 mirrors 000427 1787396269.641495 read 0x008870 thru 0x00b40d (0x2b9e bytes) 000429 1787396269.656914 MIRROR_RESYNC 5 mirrors 000430 1787396270.318931 write 0x035b25 thru 0x03e26d (0x8749 bytes) 000432 1787396270.395562 mapread 0x02ebc9 thru 0x039a01 (0xae39 bytes) 000433 1787396270.423586 trunc from 0x03e26e to 0x02fa25 (0xe848 bytes) 000434 1787396270.506476 MIRROR_RESYNC 5 mirrors 000435 1787396271.847853 read 0x015f8d thru 0x025cca (0xfd3e bytes) 000436 1787396271.859801 write 0x02aff6 thru 0x038e33 (0xde3e bytes) 000437 1787396271.985134 MIRROR_RESYNC 5 mirrors 000439 1787396273.382145 mapread 0x031b7f thru 0x038e33 (0x72b5 bytes) 000440 1787396273.402966 read 0x00895e thru 0x00d212 (0x48b5 bytes) 000441 1787396273.418380 trunc from 0x038e34 to 0x031f6a (0x6ec9 bytes) 000444 1787396273.570369 mapread 0x025756 thru 0x027a1f (0x22ca bytes) 000447 1787396273.614114 MIRROR_RESYNC 5 mirrors 000448 1787396274.799409 mapwrite 0x001b05 thru 0x01129c (0xf798 bytes) 000449 1787396275.039304 write 0x03a0ae thru 0x03ffff (0x5f52 bytes) 000452 1787396275.048108 mapwrite 0x023288 thru 0x02593f (0x26b8 bytes) 000453 1787396275.141977 read 0x02b19c thru 0x035b10 (0xa975 bytes) 000454 1787396275.149865 write 0x003d23 thru 0x010c0d (0xceeb bytes) 000455 1787396275.152061 read 0x018d74 thru 0x01d2df (0x456c bytes) 000456 1787396275.168967 read 0x037917 thru 0x03ffff (0x86e9 bytes) 000457 1787396275.172537 mapread 0x02a856 thru 0x032d72 (0x851d bytes) truncating to largest ever: 0x3e361 000458 1787396275.176858 trunc from 0x040000 to 0x03e360 (0x1c9f bytes) 000459 1787396275.213984 write 0x035357 thru 0x03ffff (0xaca9 bytes) 000460 1787396275.216090 mapread 0x0144c0 thru 0x01fb2b (0xb66c bytes) 000461 1787396275.218932 read 0x0138aa thru 0x014e48 (0x159f bytes) 000462 1787396275.220120 write 0x015a0e thru 0x01b1a3 (0x5796 bytes) 000465 1787396275.224857 trunc from 0x040000 to 0x0366b4 (0x994b bytes) 000466 1787396275.259385 read 0x01a3f9 thru 0x028d28 (0xe930 bytes) 000467 1787396275.261430 trunc from 0x0366b5 to 0x03827e (0x1bca bytes) truncating to largest ever: 0x3fb22 000468 1787396275.288696 trunc from 0x03827f to 0x03fb21 (0x78a3 bytes) 000469 1787396275.308367 read 0x02f996 thru 0x02fbad (0x218 bytes) 000471 1787396275.312861 mapread 0x03f387 thru 0x03fb21 (0x79b bytes) 000472 1787396275.325712 read 0x03200c thru 0x03fb21 (0xdb16 bytes) 000474 1787396275.328361 mapwrite 0x00082d thru 0x005c93 (0x5467 bytes) 000478 1787396275.385355 trunc from 0x03fb22 to 0x02d05c (0x12ac5 bytes) 000479 1787396275.410871 mapread 0x01a61a thru 0x02907d (0xea64 bytes) 000480 1787396275.416557 trunc from 0x02d05d to 0x03e480 (0x11424 bytes) 000481 1787396275.450239 write 0x03bc98 thru 0x03ffff (0x4368 bytes) 000482 1787396275.466599 read 0x033a85 thru 0x038880 (0x4dfc bytes) 000483 1787396275.475615 mapwrite 0x00b5c2 thru 0x016636 (0xb075 bytes) 000486 1787396275.619680 read 0x0075aa thru 0x014da5 (0xd7fc bytes) 000487 1787396275.626901 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeae18a2e (stale) Mirror 7: 0xeae18a2e (stale) Mirror 11: 0xeae18a2e (stale) Mirror 13: 0xeae18a2e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb9e74f8f Mirror 14: 0xb9e74f8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000488 1787396276.886953 write 0x02e1aa thru 0x036cef (0x8b46 bytes) 000489 1787396277.027886 trunc from 0x040000 to 0x02c84c (0x137b3 bytes) 000491 1787396277.051992 read 0x01bb08 thru 0x01ec74 (0x316d bytes) 000492 1787396277.058192 mapwrite 0x00cb7b thru 0x012588 (0x5a0e bytes) 000493 1787396277.129653 read 0x0043e5 thru 0x013e92 (0xfaae bytes) 000494 1787396277.155069 read 0x01ad07 thru 0x025898 (0xab92 bytes) 000495 1787396277.157071 write 0x0122db thru 0x020770 (0xe496 bytes) 000496 1787396277.159877 write 0x00ff12 thru 0x01833d (0x842c bytes) 000497 1787396277.161522 trunc from 0x02c84d to 0x01357b (0x192d1 bytes) 000499 1787396277.186883 mapwrite 0x0207e9 thru 0x02f78e (0xefa6 bytes) 000500 1787396277.268649 read 0x01c6d3 thru 0x027104 (0xaa32 bytes) 000501 1787396277.270051 read 0x01b04b thru 0x028b0b (0xdac1 bytes) 000505 1787396277.273375 mapread 0x0099d1 thru 0x00f2dd (0x590d bytes) 000507 1787396277.275913 mapwrite 0x0181c5 thru 0x0264c4 (0xe300 bytes) 000508 1787396277.317564 read 0x02e0f3 thru 0x02f78e (0x169c bytes) 000510 1787396277.318957 mapread 0x012581 thru 0x01dded (0xb86d bytes) 000511 1787396277.321431 trunc from 0x02f78f to 0x0020f1 (0x2d69d bytes) 000513 1787396277.350456 mapread 0x00013e thru 0x0020f1 (0x1fb4 bytes) 000514 1787396277.352604 write 0x011806 thru 0x01b3bf (0x9bba bytes) 000516 1787396277.367476 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b3c0): Mirror 5: 0xbe1080ba (stale) Mirror 7: 0xbe1080ba (stale) Mirror 11: 0xbe1080ba (stale) Mirror 13: 0xbe1080ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b3c0): Mirror 5: 0xbe1080ba (stale) Mirror 7: 0xbe1080ba (stale) Mirror 11: 0xbe1080ba (stale) Mirror 13: 0xbe1080ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b3c0: skipped 000517 1787396278.052056 read 0x000ba4 thru 0x006620 (0x5a7d bytes) 000520 1787396278.069489 write 0x014a69 thru 0x0216ea (0xcc82 bytes) 000522 1787396278.085747 trunc from 0x0216eb to 0x02d35b (0xbc71 bytes) 000523 1787396278.130550 read 0x022f0d thru 0x02c024 (0x9118 bytes) 000524 1787396278.158774 mapwrite 0x000f72 thru 0x010c4a (0xfcd9 bytes) 000528 1787396278.271692 write 0x011944 thru 0x015851 (0x3f0e bytes) 000532 1787396278.279618 trunc from 0x02d35c to 0x00ba88 (0x218d3 bytes) 000533 1787396278.302315 mapread 0x00b52b thru 0x00ba88 (0x55e bytes) 000535 1787396278.304711 trunc from 0x00ba89 to 0x01611a (0xa692 bytes) 000536 1787396278.333842 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1611b): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1611b): Mirror 5: 0x3b83561 (stale) Mirror 7: 0x3b83561 (stale) Mirror 11: 0x3b83561 (stale) Mirror 13: 0x3b83561 (stale) CRC-32 checksum value for chunk [0, 0x1611b): Mirror 1: 0xe596f23f Mirror 14: 0xe596f23f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1611b: skipped 000537 1787396279.851808 read 0x008a2d thru 0x00f922 (0x6ef6 bytes) 000539 1787396279.894480 read 0x00eedf thru 0x014f9a (0x60bc bytes) 000540 1787396279.933478 mapwrite 0x02fcc3 thru 0x0340ba (0x43f8 bytes) 000541 1787396280.368528 trunc from 0x0340bb to 0x00929e (0x2ae1c bytes) 000542 1787396280.408530 trunc from 0x00929f to 0x00f226 (0x5f88 bytes) 000543 1787396280.469260 read 0x0022bc thru 0x00b7ed (0x9532 bytes) 000550 1787396280.517820 write 0x0166e5 thru 0x01cc56 (0x6572 bytes) 000551 1787396280.519278 mapwrite 0x03475b thru 0x036aab (0x2351 bytes) 000552 1787396280.760214 read 0x0100cf thru 0x018517 (0x8449 bytes) 000553 1787396280.772838 trunc from 0x036aac to 0x00c7af (0x2a2fc bytes) 000555 1787396280.822505 trunc from 0x00c7b0 to 0x03d6c8 (0x30f19 bytes) 000556 1787396280.871497 mapwrite 0x02e4ed thru 0x034dac (0x68c0 bytes) 000560 1787396281.034810 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d6c9): Mirror 5: 0x38bd69a0 (stale) Mirror 7: 0x38bd69a0 (stale) Mirror 11: 0x38bd69a0 (stale) Mirror 14: 0x8ce17473 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d6c9): Mirror 5: 0x38bd69a0 (stale) Mirror 7: 0x38bd69a0 (stale) Mirror 11: 0x38bd69a0 (stale) Mirror 14: 0x8ce17473 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d6c9: skipped 000562 1787396281.777504 read 0x033104 thru 0x03d6c8 (0xa5c5 bytes) 000565 1787396281.792355 trunc from 0x03d6c9 to 0x0125bf (0x2b109 bytes) 000566 1787396281.819624 write 0x0355ca thru 0x03d0c8 (0x7aff bytes) 000568 1787396281.823339 read 0x00aedd thru 0x00f7a6 (0x48ca bytes) 000571 1787396281.837074 write 0x016ce7 thru 0x01c170 (0x548a bytes) 000572 1787396281.839657 read 0x03a4c4 thru 0x03d0c8 (0x2c05 bytes) 000575 1787396281.842534 mapwrite 0x028f2c thru 0x028ffa (0xcf bytes) 000577 1787396281.935824 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3d0c9): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d0c9): Mirror 5: 0x1c347b93 (stale) Mirror 7: 0x1c347b93 (stale) Mirror 11: 0x1c347b93 (stale) Mirror 14: 0xdcaf067 (stale) CRC-32 checksum value for chunk [0, 0x3d0c9): Mirror 1: 0xfebd4077 Mirror 15: 0xfebd4077 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d0c9: skipped 000579 1787396283.405135 read 0x0356cd thru 0x038a0a (0x333e bytes) 000580 1787396283.441991 mapread 0x025345 thru 0x02ef41 (0x9bfd bytes) 000581 1787396283.509719 mapread 0x025077 thru 0x02da9d (0x8a27 bytes) 000582 1787396283.542776 write 0x03adc7 thru 0x03ffff (0x5239 bytes) 000583 1787396283.632408 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeae18a2e (stale) Mirror 7: 0xeae18a2e (stale) Mirror 11: 0xeae18a2e (stale) Mirror 15: 0xb314b0ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeae18a2e (stale) Mirror 7: 0xeae18a2e (stale) Mirror 11: 0xeae18a2e (stale) Mirror 15: 0xb314b0ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000584 1787396284.307503 write 0x000122 thru 0x005a95 (0x5974 bytes) 000585 1787396284.345145 read 0x02a108 thru 0x02f241 (0x513a bytes) 000586 1787396284.375135 mapread 0x01bb06 thru 0x0223bd (0x68b8 bytes) 000587 1787396284.416904 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeae18a2e (stale) Mirror 7: 0xeae18a2e (stale) Mirror 11: 0xeae18a2e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeae18a2e (stale) Mirror 7: 0xeae18a2e (stale) Mirror 11: 0xeae18a2e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000588 1787396285.004937 trunc from 0x040000 to 0x01a53b (0x25ac4 bytes) 000591 1787396285.035322 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1a53c): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a53c): Mirror 5: 0xa51fa410 (stale) Mirror 7: 0xa51fa410 (stale) Mirror 11: 0xa51fa410 (stale) CRC-32 checksum value for chunk [0, 0x1a53c): Mirror 1: 0x5def2d29 Mirror 12: 0x5def2d29 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a53c: skipped 000594 1787396286.735764 trunc from 0x01a53c to 0x027613 (0xd0d8 bytes) 000595 1787396286.863841 trunc from 0x027614 to 0x00b29b (0x1c378 bytes) 000596 1787396286.886770 mapread 0x00973e thru 0x00b29b (0x1b5e bytes) 000599 1787396286.906817 mapwrite 0x01a7df thru 0x0279b2 (0xd1d4 bytes) 000603 1787396287.004864 mapread 0x007831 thru 0x015bc1 (0xe391 bytes) 000605 1787396287.018772 read 0x00cc9a thru 0x01609d (0x9404 bytes) 000608 1787396287.021559 trunc from 0x0279b3 to 0x02b03f (0x368d bytes) 000610 1787396287.040612 trunc from 0x02b040 to 0x035186 (0xa147 bytes) 000611 1787396287.063333 mapwrite 0x021215 thru 0x02c745 (0xb531 bytes) 000612 1787396287.122667 write 0x011f5f thru 0x014aa3 (0x2b45 bytes) 000613 1787396287.124754 write 0x0218a1 thru 0x025324 (0x3a84 bytes) 000614 1787396287.126886 read 0x010302 thru 0x016f31 (0x6c30 bytes) 000616 1787396287.128974 mapwrite 0x038f89 thru 0x03ffff (0x7077 bytes) 000617 1787396287.316747 read 0x0091ff thru 0x010e9a (0x7c9c bytes) 000619 1787396287.321374 trunc from 0x040000 to 0x031ef5 (0xe10a bytes) 000620 1787396287.346495 read 0x027bcb thru 0x02d0d2 (0x5508 bytes) 000623 1787396287.355926 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31ef6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ef6): Mirror 5: 0xccb29287 (stale) Mirror 7: 0xccb29287 (stale) Mirror 11: 0xccb29287 (stale) Mirror 12: 0x93403785 (stale) CRC-32 checksum value for chunk [0, 0x31ef6): Mirror 1: 0xeb6e8b40 Mirror 13: 0xeb6e8b40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ef6: skipped 000625 1787396288.725310 read 0x000204 thru 0x00bafc (0xb8f9 bytes) 000627 1787396288.792874 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x31ef6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ef6): Mirror 5: 0xccb29287 (stale) Mirror 7: 0xccb29287 (stale) Mirror 11: 0xccb29287 (stale) Mirror 12: 0x93403785 (stale) CRC-32 checksum value for chunk [0, 0x31ef6): Mirror 1: 0xeb6e8b40 Mirror 13: 0xeb6e8b40 Mirror 14: 0xeb6e8b40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ef6: skipped 000628 1787396290.117051 trunc from 0x031ef6 to 0x01dfa4 (0x13f51 bytes) 000629 1787396290.287255 write 0x00db65 thru 0x011094 (0x3530 bytes) 000631 1787396290.311057 MIRROR_RESYNC 7 mirrors 000632 1787396292.086610 write 0x03e207 thru 0x03ffff (0x1df9 bytes) 000633 1787396292.231245 mapread 0x00f536 thru 0x01caca (0xd595 bytes) 000634 1787396292.344395 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x23ae2bfc (stale) Mirror 11: 0x23ae2bfc (stale) Mirror 12: 0x23ae2bfc (stale) Mirror 13: 0x23ae2bfc (stale) Mirror 14: 0x23ae2bfc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x23ae2bfc (stale) Mirror 11: 0x23ae2bfc (stale) Mirror 12: 0x23ae2bfc (stale) Mirror 13: 0x23ae2bfc (stale) Mirror 14: 0x23ae2bfc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000635 1787396293.118818 mapwrite 0x0059c0 thru 0x0103d1 (0xaa12 bytes) 000636 1787396293.276680 trunc from 0x040000 to 0x027b04 (0x184fb bytes) 000637 1787396293.319378 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x27b05): Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b05): Mirror 5: 0xf355fd3b (stale) Mirror 11: 0xf355fd3b (stale) Mirror 12: 0xf355fd3b (stale) Mirror 13: 0xf355fd3b (stale) Mirror 14: 0xf355fd3b (stale) CRC-32 checksum value for chunk [0, 0x27b05): Mirror 1: 0xe1a7f8a Mirror 15: 0xe1a7f8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b05: skipped 000638 1787396294.537265 mapread 0x025ca8 thru 0x027b04 (0x1e5d bytes) 000639 1787396294.566356 read 0x0135fe thru 0x01ed52 (0xb755 bytes) 000640 1787396294.583468 mapwrite 0x00fae7 thru 0x01f245 (0xf75f bytes) 000641 1787396294.904597 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x27b05): Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b05): Mirror 5: 0xf355fd3b (stale) Mirror 11: 0xf355fd3b (stale) Mirror 12: 0xf355fd3b (stale) Mirror 13: 0xf355fd3b (stale) Mirror 14: 0xf355fd3b (stale) Mirror 15: 0xe1a7f8a (stale) CRC-32 checksum value for chunk [0, 0x27b05): Mirror 1: 0xf35d790e Mirror 16: 0xf35d790e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b05: skipped 000644 1787396296.679775 write 0x03b90e thru 0x03e1b0 (0x28a3 bytes) 000645 1787396296.802174 mapwrite 0x02640e thru 0x02bc8c (0x587f bytes) 000646 1787396297.030233 mapwrite 0x027ff2 thru 0x02ca3d (0x4a4c bytes) 000647 1787396297.091257 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e1b1): Mirror 5: 0xf78327ee (stale) Mirror 11: 0xf78327ee (stale) Mirror 12: 0xf78327ee (stale) Mirror 14: 0xf78327ee (stale) Mirror 15: 0xfe18e220 (stale) Mirror 16: 0x67f20b6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1b1): Mirror 5: 0xf78327ee (stale) Mirror 11: 0xf78327ee (stale) Mirror 12: 0xf78327ee (stale) Mirror 14: 0xf78327ee (stale) Mirror 15: 0xfe18e220 (stale) Mirror 16: 0x67f20b6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1b1: skipped 000649 1787396297.932576 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3e1b1): Mirror 5: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1b1): Mirror 5: 0xf78327ee (stale) Mirror 11: 0xf78327ee (stale) Mirror 12: 0xf78327ee (stale) Mirror 14: 0xf78327ee (stale) Mirror 15: 0xfe18e220 (stale) Mirror 16: 0x67f20b6d (stale) CRC-32 checksum value for chunk [0, 0x3e1b1): Mirror 1: 0x7780273f Mirror 17: 0x7780273f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1b1: skipped 000650 1787396299.343005 mapread 0x017a30 thru 0x02141c (0x99ed bytes) 000651 1787396299.409228 mapwrite 0x02715b thru 0x02727d (0x123 bytes) 000655 1787396299.660309 write 0x013672 thru 0x021081 (0xda10 bytes) 000656 1787396299.668267 trunc from 0x03e1b1 to 0x03f0cd (0xf1d bytes) 000659 1787396299.698609 MIRROR_RESYNC 8 mirrors 000660 1787396301.454243 read 0x030ddc thru 0x030fcc (0x1f1 bytes) 000661 1787396301.461027 read 0x03b7e3 thru 0x03f0cd (0x38eb bytes) 000662 1787396301.498212 trunc from 0x03f0ce to 0x00b808 (0x338c5 bytes) 000663 1787396301.643843 mapread 0x006126 thru 0x00b808 (0x56e3 bytes) 000665 1787396301.704558 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb809): Mirror 11: 0xdaed9af0 (stale) Mirror 12: 0xdaed9af0 (stale) Mirror 14: 0xdaed9af0 (stale) Mirror 15: 0xdaed9af0 (stale) Mirror 16: 0xdaed9af0 (stale) Mirror 17: 0xdaed9af0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb809): Mirror 11: 0xdaed9af0 (stale) Mirror 12: 0xdaed9af0 (stale) Mirror 14: 0xdaed9af0 (stale) Mirror 15: 0xdaed9af0 (stale) Mirror 16: 0xdaed9af0 (stale) Mirror 17: 0xdaed9af0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb809: skipped 000666 1787396302.352880 write 0x032a43 thru 0x03b41e (0x89dc bytes) 000669 1787396302.365576 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b41f): Mirror 11: 0xfbd3de7 (stale) Mirror 14: 0xfbd3de7 (stale) Mirror 15: 0xfbd3de7 (stale) Mirror 16: 0xfbd3de7 (stale) Mirror 17: 0xfbd3de7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b41f): Mirror 11: 0xfbd3de7 (stale) Mirror 14: 0xfbd3de7 (stale) Mirror 15: 0xfbd3de7 (stale) Mirror 16: 0xfbd3de7 (stale) Mirror 17: 0xfbd3de7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b41f: skipped 000670 1787396303.189971 mapread 0x030883 thru 0x03a357 (0x9ad5 bytes) 000671 1787396303.217455 write 0x029f82 thru 0x03556d (0xb5ec bytes) 000672 1787396303.234917 mapread 0x0178f4 thru 0x01b09f (0x37ac bytes) 000673 1787396303.269215 trunc from 0x03b41f to 0x00ff5f (0x2b4bf bytes) 000674 1787396303.334950 read 0x00c58c thru 0x00ff5f (0x39d4 bytes) 000677 1787396303.358270 write 0x013275 thru 0x019da5 (0x6b31 bytes) 000678 1787396303.361873 write 0x03566f thru 0x03ba57 (0x63e9 bytes) 000679 1787396303.363405 mapread 0x03147e thru 0x03ba57 (0xa5da bytes) 000681 1787396303.385421 trunc from 0x03ba58 to 0x03c753 (0xcfc bytes) 000682 1787396303.421443 read 0x00ec3f thru 0x018451 (0x9813 bytes) 000683 1787396303.448526 mapread 0x01a663 thru 0x01b1c3 (0xb61 bytes) 000685 1787396303.451416 read 0x01c162 thru 0x02a685 (0xe524 bytes) 000693 1787396303.462599 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c754): Mirror 11: 0x349ae6ec (stale) Mirror 14: 0x349ae6ec (stale) Mirror 15: 0x349ae6ec (stale) Mirror 17: 0x349ae6ec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c754): Mirror 11: 0x349ae6ec (stale) Mirror 14: 0x349ae6ec (stale) Mirror 15: 0x349ae6ec (stale) Mirror 17: 0x349ae6ec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c754: skipped 000695 1787396304.018049 trunc from 0x03c754 to 0x0025ed (0x3a166 bytes) 000696 1787396304.036936 mapwrite 0x017ac5 thru 0x0197b8 (0x1cf4 bytes) 000697 1787396304.148391 read 0x012c88 thru 0x0179e5 (0x4d5e bytes) 000698 1787396304.149250 write 0x035141 thru 0x03f79f (0xa65f bytes) 000699 1787396304.156651 trunc from 0x03f7a0 to 0x003317 (0x3c488 bytes) 000700 1787396304.211060 trunc from 0x003318 to 0x006e04 (0x3aed bytes) 000701 1787396304.234003 mapwrite 0x03c71e thru 0x03ffff (0x38e2 bytes) 000702 1787396304.371994 trunc from 0x040000 to 0x015f11 (0x2a0ee bytes) 000705 1787396304.421434 MIRROR_RESYNC 5 mirrors 000706 1787396305.876736 MIRROR_RESYNC 5 mirrors 000707 1787396306.338340 trunc from 0x015f12 to 0x02eb84 (0x18c73 bytes) 000708 1787396306.439382 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2eb85): Mirror 14: 0x6fee805a (stale) Mirror 15: 0x6fee805a (stale) Mirror 17: 0x6fee805a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eb85): Mirror 14: 0x6fee805a (stale) Mirror 15: 0x6fee805a (stale) Mirror 17: 0x6fee805a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eb85: skipped 000709 1787396307.164365 mapread 0x00bff1 thru 0x010b3f (0x4b4f bytes) 000710 1787396307.209545 read 0x000975 thru 0x001b95 (0x1221 bytes) 000711 1787396307.215461 read 0x009088 thru 0x015038 (0xbfb1 bytes) 000715 1787396307.219104 mapwrite 0x039a73 thru 0x03fe33 (0x63c1 bytes) 000717 1787396307.345930 MIRROR_RESYNC 4 mirrors 000718 1787396308.613032 read 0x00ac96 thru 0x019755 (0xeac0 bytes) 000719 1787396308.628065 mapread 0x03ec33 thru 0x03fe33 (0x1201 bytes) 000720 1787396308.663318 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) 000721 1787396309.111012 mapread 0x019a3a thru 0x01d742 (0x3d09 bytes) 000722 1787396309.147437 mapread 0x03de92 thru 0x03fe33 (0x1fa2 bytes) 000723 1787396309.168411 MIRROR_EXTEND to 4 mirrors 000726 1787396309.845460 read 0x017921 thru 0x01bf0b (0x45eb bytes) 000729 1787396309.854452 read 0x01f9e1 thru 0x02f781 (0xfda1 bytes) 000732 1787396309.875668 read 0x03ddc9 thru 0x03fe33 (0x206b bytes) 000733 1787396309.877917 mapwrite 0x033245 thru 0x03557d (0x2339 bytes) 000734 1787396310.016007 mapread 0x02f8d7 thru 0x039b1d (0xa247 bytes) 000735 1787396310.018716 mapread 0x00a2f4 thru 0x00a929 (0x636 bytes) 000738 1787396310.022502 trunc from 0x03fe34 to 0x00c043 (0x33df0 bytes) 000739 1787396310.045528 read 0x00a451 thru 0x00c043 (0x1bf3 bytes) 000740 1787396310.046383 trunc from 0x00c044 to 0x030c90 (0x24c4d bytes) 000741 1787396310.069243 mapwrite 0x02c566 thru 0x02c5d5 (0x70 bytes) 000742 1787396310.136234 write 0x02993d thru 0x033584 (0x9c48 bytes) 000743 1787396310.139109 mapwrite 0x0295dc thru 0x0393a3 (0xfdc8 bytes) 000745 1787396310.220528 mapread 0x031db3 thru 0x0393a3 (0x75f1 bytes) 000747 1787396310.228622 MIRROR_RESYNC 4 mirrors 000748 1787396311.050967 read 0x0162f4 thru 0x01ef7d (0x8c8a bytes) 000751 1787396311.066613 MIRROR_EXTEND to 5 mirrors 000752 1787396311.941338 read 0x029e7d thru 0x037b0a (0xdc8e bytes) 000753 1787396311.967889 write 0x037b5b thru 0x039d66 (0x220c bytes) 000754 1787396312.111802 read 0x01c679 thru 0x01f8d9 (0x3261 bytes) 000756 1787396312.134976 write 0x0394c2 thru 0x03ffff (0x6b3e bytes) 000757 1787396312.145768 MIRROR_RESYNC 5 mirrors 000758 1787396313.492237 read 0x03441d thru 0x03f22a (0xae0e bytes) 000760 1787396313.508189 mapread 0x036abc thru 0x03ae7a (0x43bf bytes) 000761 1787396313.509245 mapwrite 0x0260db thru 0x02da8b (0x79b1 bytes) 000764 1787396313.721997 write 0x02486f thru 0x02ad4e (0x64e0 bytes) 000765 1787396313.724302 write 0x0117d8 thru 0x01a94d (0x9176 bytes) 000766 1787396313.747052 read 0x02bb48 thru 0x0316da (0x5b93 bytes) 000767 1787396313.767090 write 0x00c502 thru 0x01aa65 (0xe564 bytes) 000769 1787396313.776195 MIRROR_RESYNC 5 mirrors 000770 1787396315.206974 MIRROR_RESYNC 5 mirrors 000772 1787396315.949955 write 0x005286 thru 0x0092f0 (0x406b bytes) 000773 1787396316.080397 trunc from 0x040000 to 0x0377b0 (0x884f bytes) 000774 1787396316.103912 read 0x01f6e3 thru 0x02aba7 (0xb4c5 bytes) 000775 1787396316.113655 mapread 0x029d52 thru 0x035604 (0xb8b3 bytes) 000777 1787396316.149462 trunc from 0x0377b1 to 0x00276f (0x35041 bytes) 000778 1787396316.187329 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2770): Mirror 15: 0xaca299b3 (stale) Mirror 16: 0xaca299b3 (stale) Mirror 17: 0xaca299b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2770): Mirror 15: 0xaca299b3 (stale) Mirror 16: 0xaca299b3 (stale) Mirror 17: 0xaca299b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2770: skipped truncating to largest ever: 0x3fbb9 000779 1787396316.615092 trunc from 0x002770 to 0x03fbb8 (0x3d449 bytes) 000781 1787396316.642492 trunc from 0x03fbb9 to 0x020060 (0x1fb58 bytes) 000783 1787396316.669389 mapwrite 0x004fa6 thru 0x006493 (0x14ee bytes) 000784 1787396316.781357 mapread 0x01cacc thru 0x020060 (0x3595 bytes) 000786 1787396316.802271 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20061): Mirror 16: 0xa4377eaf (stale) Mirror 17: 0xa4377eaf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20061): Mirror 16: 0xa4377eaf (stale) Mirror 17: 0xa4377eaf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20061: skipped 000787 1787396317.206574 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x20061): Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20061): Mirror 16: 0xa4377eaf (stale) Mirror 17: 0xa4377eaf (stale) CRC-32 checksum value for chunk [0, 0x20061): Mirror 1: 0x584b77ee Mirror 18: 0x584b77ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20061: skipped 000788 1787396318.276849 mapread 0x01f447 thru 0x020060 (0xc1a bytes) 000789 1787396318.337577 mapwrite 0x01a172 thru 0x02697e (0xc80d bytes) 000790 1787396318.609441 mapwrite 0x00120d thru 0x00200d (0xe01 bytes) 000794 1787396318.707673 trunc from 0x02697f to 0x018760 (0xe21e bytes) 000795 1787396318.745932 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18761): Mirror 17: 0xc572a086 (stale) Mirror 18: 0x400fba18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18761): Mirror 17: 0xc572a086 (stale) Mirror 18: 0x400fba18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18761: skipped 000798 1787396319.061702 mapread 0x00b9e8 thru 0x00c184 (0x79d bytes) 000799 1787396319.071673 trunc from 0x018761 to 0x03373d (0x1afdd bytes) 000800 1787396319.091353 mapread 0x00ff99 thru 0x014fa5 (0x500d bytes) 000801 1787396319.102371 mapread 0x0058a1 thru 0x00fd6f (0xa4cf bytes) 000804 1787396319.116818 mapwrite 0x0088b4 thru 0x00b35e (0x2aab bytes) 000805 1787396319.164535 trunc from 0x03373e to 0x029ff0 (0x974d bytes) 000806 1787396319.182649 read 0x010e11 thru 0x0143b1 (0x35a1 bytes) 000807 1787396319.188082 MIRROR_RESYNC 3 mirrors 000808 1787396320.014050 mapwrite 0x03933f thru 0x03faca (0x678c bytes) 000810 1787396320.224362 write 0x03ca40 thru 0x03ffff (0x35c0 bytes) 000811 1787396320.230045 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x92d93b6 (stale) Mirror 18: 0x92d93b6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x592d607d Mirror 19: 0x592d607d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000816 1787396321.064431 write 0x01cf34 thru 0x02b4e4 (0xe5b1 bytes) 000818 1787396321.164313 write 0x00f836 thru 0x01655a (0x6d25 bytes) 000822 1787396321.177363 read 0x019104 thru 0x01fe50 (0x6d4d bytes) 000823 1787396321.182821 trunc from 0x040000 to 0x033b2b (0xc4d4 bytes) 000825 1787396321.250398 mapwrite 0x018bcd thru 0x026a43 (0xde77 bytes) 000827 1787396321.380058 write 0x019d1e thru 0x0285c2 (0xe8a5 bytes) 000831 1787396321.392178 mapread 0x020efb thru 0x02eabb (0xdbc1 bytes) 000832 1787396321.412279 trunc from 0x033b2c to 0x01c653 (0x174d8 bytes) 000833 1787396321.445043 write 0x00f633 thru 0x01a25c (0xac2a bytes) 000834 1787396321.450321 read 0x000355 thru 0x001d9a (0x1a46 bytes) 000835 1787396321.486824 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c654): Mirror 17: 0xd7614445 (stale) Mirror 18: 0xd7614445 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c654): Mirror 17: 0xd7614445 (stale) Mirror 18: 0xd7614445 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c654: skipped 000838 1787396322.053641 read 0x018185 thru 0x01c653 (0x44cf bytes) 000840 1787396322.070751 mapwrite 0x02e7b3 thru 0x03dd33 (0xf581 bytes) 000842 1787396322.235525 mapwrite 0x0050da thru 0x00dacc (0x89f3 bytes) 000845 1787396322.339298 read 0x017a15 thru 0x01cd99 (0x5385 bytes) 000846 1787396322.371398 write 0x025599 thru 0x032f66 (0xd9ce bytes) 000847 1787396322.372832 mapwrite 0x02f551 thru 0x037cd7 (0x8787 bytes) 000848 1787396322.464050 mapwrite 0x00c41c thru 0x018231 (0xbe16 bytes) 000849 1787396322.568334 write 0x0218a9 thru 0x02f6ad (0xde05 bytes) 000850 1787396322.572027 mapread 0x01ba73 thru 0x02320b (0x7799 bytes) 000851 1787396322.590801 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dd34): Mirror 18: 0xab979422 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd34): Mirror 18: 0xab979422 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd34: skipped 000852 1787396323.167411 write 0x017907 thru 0x020d35 (0x942f bytes) 000853 1787396323.201186 read 0x015436 thru 0x016e93 (0x1a5e bytes) 000854 1787396323.211139 mapread 0x01cfee thru 0x02634a (0x935d bytes) 000855 1787396323.245067 trunc from 0x03dd34 to 0x00e4f7 (0x2f83c bytes) 000857 1787396323.262311 read 0x003ff0 thru 0x00e4f7 (0xa508 bytes) 000859 1787396323.274613 read 0x004bef thru 0x00ac52 (0x6064 bytes) 000860 1787396323.275309 read 0x004653 thru 0x009905 (0x52b3 bytes) 000861 1787396323.280831 MIRROR_RESYNC 2 mirrors 000863 1787396324.019566 mapwrite 0x0209ec thru 0x028620 (0x7c35 bytes) 000864 1787396324.280573 read 0x00bfa3 thru 0x00d2f9 (0x1357 bytes) 000865 1787396324.290556 read 0x011047 thru 0x0143b9 (0x3373 bytes) 000866 1787396324.310453 mapread 0x027b9e thru 0x028620 (0xa83 bytes) 000872 1787396324.320292 write 0x004749 thru 0x00c143 (0x79fb bytes) 000873 1787396324.324900 mapwrite 0x03f6b2 thru 0x03ffff (0x94e bytes) 000874 1787396324.427876 write 0x0176c7 thru 0x01d4e5 (0x5e1f bytes) 000876 1787396324.431566 mapwrite 0x019591 thru 0x022434 (0x8ea4 bytes) 000877 1787396324.522973 MIRROR_RESYNC 2 mirrors 000878 1787396325.250451 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x198:0x0]: Invalid argument (22) 000880 1787396325.426716 mapread 0x021975 thru 0x0294c4 (0x7b50 bytes) 000882 1787396325.468813 read 0x02e4f6 thru 0x03d447 (0xef52 bytes) 000883 1787396325.485141 read 0x024bac thru 0x03196f (0xcdc4 bytes) 000884 1787396325.505261 write 0x030001 thru 0x03bd3f (0xbd3f bytes) 000885 1787396325.555999 read 0x028e98 thru 0x036398 (0xd501 bytes) 000886 1787396325.576436 mapwrite 0x002e60 thru 0x004444 (0x15e5 bytes) 000887 1787396325.650649 mapwrite 0x02fb80 thru 0x037830 (0x7cb1 bytes) 000889 1787396325.704108 trunc from 0x040000 to 0x030f94 (0xf06b bytes) 000890 1787396325.746040 trunc from 0x030f95 to 0x00a5bd (0x269d7 bytes) 000891 1787396325.764718 write 0x039972 thru 0x03ffff (0x668e bytes) 000892 1787396325.766094 mapread 0x000173 thru 0x00b7c4 (0xb652 bytes) 000893 1787396325.772571 read 0x02db0b thru 0x03c6e9 (0xebdf bytes) 000895 1787396325.781225 write 0x013460 thru 0x01a4db (0x707c bytes) 000896 1787396325.782895 write 0x01fd82 thru 0x02a1f8 (0xa477 bytes) 000901 1787396325.788061 trunc from 0x040000 to 0x03bfe0 (0x401f bytes) 000908 1787396325.818930 write 0x027009 thru 0x031d24 (0xad1c bytes) 000909 1787396325.820550 trunc from 0x03bfe1 to 0x00e9f9 (0x2d5e7 bytes) 000910 1787396325.837803 mapread 0x00a175 thru 0x00e9f9 (0x4885 bytes) 000914 1787396325.842935 mapwrite 0x0199eb thru 0x01fb8f (0x61a5 bytes) 000915 1787396325.950981 trunc from 0x01fb90 to 0x02fda2 (0x10213 bytes) 000920 1787396325.979598 read 0x028c3c thru 0x02ecb1 (0x6076 bytes) 000922 1787396326.003605 write 0x039fc7 thru 0x03ffff (0x6039 bytes) 000933 1787396326.015564 read 0x014977 thru 0x020486 (0xbb10 bytes) 000937 1787396326.021447 write 0x03fbff thru 0x03ffff (0x401 bytes) 000938 1787396326.022775 trunc from 0x040000 to 0x00378f (0x3c870 bytes) 000939 1787396326.052175 write 0x039a77 thru 0x03ffff (0x6589 bytes) 000941 1787396326.054786 mapread 0x03df7e thru 0x03ffff (0x2082 bytes) 000942 1787396326.056597 write 0x026ad8 thru 0x033b3a (0xd063 bytes) 000943 1787396326.068121 write 0x03f7d5 thru 0x03ffff (0x82b bytes) 000944 1787396326.068821 read 0x0140bb thru 0x016fa3 (0x2ee9 bytes) 000945 1787396326.069479 trunc from 0x040000 to 0x0032a8 (0x3cd57 bytes) 000946 1787396326.101436 read 0x001777 thru 0x0032a8 (0x1b32 bytes) 000948 1787396326.104811 trunc from 0x0032a9 to 0x007720 (0x4478 bytes) 000949 1787396326.136908 mapwrite 0x036418 thru 0x03de42 (0x7a2b bytes) 000951 1787396326.228953 read 0x02aadf thru 0x02e6b7 (0x3bd9 bytes) 000957 1787396326.235272 write 0x007123 thru 0x0123e6 (0xb2c4 bytes) 000958 1787396326.237382 trunc from 0x03de43 to 0x00e2a0 (0x2fba2 bytes) 000965 1787396326.264967 read 0x00b7a5 thru 0x00e2a0 (0x2afc bytes) 000966 1787396326.265766 write 0x01a17e thru 0x028515 (0xe398 bytes) 000967 1787396326.268290 trunc from 0x028516 to 0x021baf (0x6966 bytes) 000968 1787396326.300385 trunc from 0x021bb0 to 0x01dcd4 (0x3edb bytes) 000969 1787396326.330131 read 0x012390 thru 0x01a978 (0x85e9 bytes) 000974 1787396326.344603 write 0x031c9e thru 0x03ef8a (0xd2ed bytes) 000975 1787396326.347017 read 0x00e142 thru 0x01dff8 (0xfeb7 bytes) 000976 1787396326.349642 mapread 0x00fd05 thru 0x01c334 (0xc630 bytes) 000977 1787396326.352503 read 0x03e8ff thru 0x03ef8a (0x68c bytes) 000978 1787396326.353748 write 0x0295a6 thru 0x02e6cd (0x5128 bytes) 000980 1787396326.367423 write 0x027ea2 thru 0x02c11e (0x427d bytes) 000981 1787396326.369178 trunc from 0x03ef8b to 0x03c0d5 (0x2eb5 bytes) 000982 1787396326.388657 mapread 0x02243b thru 0x0264af (0x4075 bytes) 000983 1787396326.392274 write 0x026589 thru 0x02aef8 (0x4970 bytes) 000985 1787396326.394958 read 0x005871 thru 0x012653 (0xcde3 bytes) 000986 1787396326.395874 read 0x037476 thru 0x03c0d5 (0x4c60 bytes) 000987 1787396326.397013 trunc from 0x03c0d6 to 0x018497 (0x23c3e bytes) 000988 1787396326.420931 write 0x030db2 thru 0x03490f (0x3b5e bytes) 000993 1787396326.429125 mapwrite 0x013c03 thru 0x01dfee (0xa3ec bytes) 000998 1787396326.553999 read 0x0028bb thru 0x002ae1 (0x227 bytes) 000999 1787396326.555123 mapwrite 0x02cf89 thru 0x03026f (0x32e7 bytes) 001000 1787396326.612000 mapwrite 0x00d5b9 thru 0x01b501 (0xdf49 bytes) All operations completed A-OK! PASS 70a (115s) == sanity-flr test 71: check mirror extend parent fid ==== 06:58:51 (1787396331) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 06:58:54 (1787396334) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.240911 s, 8.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249788 s, 4.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 06:59:04 (1787396344) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4109216 .. done Extending file size to 6993504 .. Extending file size to 7740096 .. Extending file size to 7596768 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7779648 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8684192 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9193760 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8684768 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56974 56976 56977 56979 56980 done 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) 192.168.203.103@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg303-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 07:00:29 (1787396429) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1014912 .. Extending file size to 1042208 .. failed Extending file size to 1482368 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8650048 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 3171136 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8941952 .. done Extending file size to 4516512 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6177344 .. failed Extending file size to 7250944 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8978240 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7663776 .. Extending file size to 8482208 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8606464 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58488 58489 58492 58493 58495 done 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) 192.168.203.103@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg303-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 07:01:47 (1787396507) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102606 s, 20.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.33037 s, 6.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa017e025 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 07:02:06 (1787396526) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0815462 s, 25.7 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 07:02:13 (1787396533) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1df:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 07:02:21 (1787396541) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.282636 s, 7.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 07:02:29 (1787396549) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241699 s, 8.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 07:02:37 (1787396557) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24105 s, 8.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 07:03:12 (1787396592) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132819 s, 7.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fb:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x319:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x318:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fc:0x0] } PASS 204d (62s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 07:04:14 (1787396654) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16854 s, 12.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 07:05:13 (1787396713) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg303-server mds-ost sync done. Creating new pool test_204f oleg303-server: Pool lustre.test_204f created Adding targets to pool oleg303-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg303-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.192952 s, 10.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2fe:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg303-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg303-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg303-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 07:06:27 (1787396787) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 07:06:34 (1787396794) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 07:06:40 (1787396800) oleg303-server: Pool lustre.test_206 created Adding targets to pool oleg303-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg303-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg303-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg303-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg303-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 07:07:02 (1787396822) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830884 s, 12.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377938 s, 27.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb16f23e9 Mirror 3: 0xb16f23e9 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 07:07:12 (1787396832) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 07:07:15 (1787396835) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 07:07:17 (1787396837) Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.122529 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2520000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2520000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2520000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2370000 Read this (4 bytes): BBBB 192.168.203.103@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg303-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 07:07:50 (1787396870) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0208786 s, 392 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 07:07:56 (1787396876) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681735 s, 15.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x204 lmm_fid: [0x200000404:0x204:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x204 lmm_fid: [0x200000404:0x204:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834436 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 07:07:58.000000000 -0400 Modify: 2026-08-22 07:07:58.000000000 -0400 Change: 2026-08-22 07:07:59.000000000 -0400 Birth: 2026-08-22 07:07:58.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 07:08:04 (1787396884) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 07:08:08 (1787396888) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.729768 s, 56.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0743139 s, 55.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 22 07:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x206:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 22 07:08 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 07:08:17 (1787396897) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 07:08:24 (1787396904) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1780 sec =========== 07:08:33 (1787396913) === sanity-flr: start cleanup 07:08:34 (1787396914) === === sanity-flr: finish cleanup 07:08:41 (1787396921) ===