-----============= acceptance-small: sanity-flr ============----- Sat Aug 22 01:57:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg148-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:57:25 (1787378245) === oleg148-client.virtnet: executing check_config_client /mnt/lustre oleg148-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg148-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8bd044c84000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8bd044c84000.idle_timeout=debug disable quota as required oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:58:05 (1787378285) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7588] == sanity-flr test 0a: lfs mirror create with -N option == 01:58:07 (1787378287) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:58:21 (1787378301) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:58:23 (1787378303) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:58:26 (1787378306) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:58:47 (1787378327) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:58:50 (1787378330) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:58:54 (1787378334) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8bd044c84000.stats=clear osc.lustre-OST0001-osc-ffff8bd044c84000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:59:02 (1787378342) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:59:11 (1787378351) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:59:20 (1787378360) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:59:23 (1787378363) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:59:31 (1787378371) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 01:59:39 (1787378379) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:59:47 (1787378387) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:59:51 (1787378391) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:59:58 (1787378398) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 02:00:09 (1787378409) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 02:00:16 (1787378416) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.34999 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793213 s, 13.2 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 02:00:27 (1787378427) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.416171 s, 47.9 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 02:00:34 (1787378434) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 02:00:43 (1787378443) Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bd044c84000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bd044c84000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg148-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bd044c84000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bd044c84000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (129s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 02:02:52 (1787378572) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.355727 s, 23.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bd044c84000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bd044c84000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 02:03:36 (1787378616) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bd044c84000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bd044c84000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bd044c84000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bd044c84000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:04:56 (1787378696) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.92766 s, 13.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 22 02:05 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.173s user 0m0.000s sys 0m0.180s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8bd044c84000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8bd044c84000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.333s user 0m0.002s sys 0m0.204s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8bd044c84000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bd044c84000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.179s user 0m0.001s sys 0m0.179s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8bd044c84000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8bd044c84000.ost_server_uuid in FULL state after 0 sec PASS 33b (84s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:06:20 (1787378780) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:06:24 (1787378784) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:06:27 (1787378787) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 02:06:32 (1787378792) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 02:06:39 (1787378799) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.546133 s, 23.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:06:51 (1787378811) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524495 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314309 s, 33.4 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:06:57 (1787378817) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:07:04 (1787378824) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00224845 s, 455 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00119097 s, 860 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00100053 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00120373 s, 851 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 02:07:14 (1787378834) create test files with size 15738361 .. 06aa4af653bbdc935530f145c748df89 - 29d4c640c503e009ec1a5ce69c5cd27c - 9ea3cf68829343d3f74de97e79a1a26d - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 02:07:35 (1787378855) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:07:46 (1787378866) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:07:52 (1787378872) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100007 s, 10.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368027 s, 28.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xda:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 02:08:00 (1787378880) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229625 s, 9.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.358547 s, 11.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25282 s, 8.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 02:08:16 (1787378896) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 02:08:19 (1787378899) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:08:22 (1787378902) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238142 s, 44.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 02:08:30 (1787378910) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:08:33 (1787378913) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xbf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:08:43 (1787378923) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.295065 s, 35.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:08:51 (1787378931) llite.lustre-ffff8bd044bf6000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.258732 s, 40.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8bd044a83000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:09:04 (1787378944) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:09:43 (1787378983) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:10:15 (1787379015) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 02:10:18 (1787379018) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.300733 s, 10.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.108774 s, 28.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6cefbb19 (nosync,stale) Mirror 2: 0x6cefbb19 (nosync,stale) ** verify mirror 2 doesn't change original checksum: cc470e817aaee65b9c07930f68e1376f - mirror 1 checksum: 53e3c3409c624bd1d8204d057a1e3fa7 - mirror 2 checksum: cc470e817aaee65b9c07930f68e1376f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T02:10:21-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-22T02:10:21-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: cc470e817aaee65b9c07930f68e1376f - mirror 1 checksum: 53e3c3409c624bd1d8204d057a1e3fa7 - mirror 2 checksum: 53e3c3409c624bd1d8204d057a1e3fa7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T02:10:21-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd5:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-22T02:10:21-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:10:31 (1787379031) Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 02:10:37 (1787379037) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 02:10:59 (1787379059) targets are mounted 02:10:59 (1787379059) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296674 s, 35.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd7:0x0] } PASS 50A (44s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:11:15 (1787379075) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:11:19 (1787379079) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:11:23 (1787379083) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00649045 s, 2.5 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:11:32 (1787379092) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:11:35 (1787379095) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:11:39 (1787379099) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (18s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:11:57 (1787379117) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:12:26 (1787379146) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:12:51 (1787379171) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:13:17 (1787379197) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 02:13:27 (1787379207) mirror create pid 53759 mirror split pid 53761 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53759 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 53761 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 02:14:34 (1787379274) Seed set to 9276 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x24afc 000002 1787379276.410799 trunc from 00000000 to 0x024afb (0x24afc bytes) truncating to largest ever: 0x2528e 000005 1787379276.432711 trunc from 0x024afc to 0x02528d (0x792 bytes) 000006 1787379276.455262 mapwrite 0x01942c thru 0x02216a (0x8d3f bytes) 000007 1787379276.565139 mapwrite 0x0252ce thru 0x03391e (0xe651 bytes) 000009 1787379276.645798 mapread 0x027b1f thru 0x03391e (0xbe00 bytes) truncating to largest ever: 0x2e7a9 000011 1787379276.652593 trunc from 0x03391f to 0x02e7a8 (0x5176 bytes) 000012 1787379276.672713 trunc from 0x02e7a9 to 0x00a66e (0x2413a bytes) 000014 1787379276.690751 read 0x001f95 thru 0x00a66e (0x86da bytes) 000016 1787379276.700206 trunc from 0x00a66f to 0x0020da (0x8594 bytes) 000021 1787379276.718168 mapread 0x001e2a thru 0x0020da (0x2b1 bytes) 000022 1787379276.719504 mapwrite 0x028773 thru 0x029cc3 (0x1551 bytes) 000023 1787379276.782596 write 0x00a594 thru 0x00e08b (0x3af8 bytes) 000024 1787379276.796782 mapread 0x014d7c thru 0x01e60d (0x9892 bytes) 000026 1787379276.817409 mapread 0x00d74b thru 0x011827 (0x40dd bytes) 000029 1787379276.820863 write 0x01641f thru 0x01ba1f (0x5601 bytes) 000030 1787379276.822524 write 0x02c9bf thru 0x038c03 (0xc245 bytes) 000032 1787379276.825847 mapwrite 0x035a40 thru 0x038021 (0x25e2 bytes) 000033 1787379276.884457 write 0x00fff5 thru 0x016ea4 (0x6eb0 bytes) 000034 1787379276.885792 mapread 0x02a5a6 thru 0x032f33 (0x898e bytes) 000040 1787379276.897181 mapread 0x0112e0 thru 0x01bec1 (0xabe2 bytes) 000041 1787379276.899186 mapwrite 0x03880f thru 0x03e722 (0x5f14 bytes) 000042 1787379276.965619 trunc from 0x03e723 to 0x001865 (0x3cebd bytes) 000044 1787379276.986088 mapwrite 0x03318d thru 0x0368d3 (0x3747 bytes) 000045 1787379277.051916 read 0x01a124 thru 0x0252dc (0xb1b9 bytes) 000046 1787379277.062949 mapwrite 0x02fecf thru 0x037ca1 (0x7dd3 bytes) 000048 1787379277.144769 write 0x032e8e thru 0x03912f (0x62a2 bytes) 000049 1787379277.146660 trunc from 0x039130 to 0x024215 (0x14f1a bytes) 000050 1787379277.165108 mapwrite 0x0231f8 thru 0x0316ae (0xe4b7 bytes) 000051 1787379277.308716 mapwrite 0x01d9d7 thru 0x01fa4b (0x2075 bytes) 000053 1787379277.392908 mapwrite 0x0119b3 thru 0x01ed91 (0xd3df bytes) 000054 1787379277.453898 write 0x020f70 thru 0x0232bc (0x234d bytes) 000056 1787379277.458432 trunc from 0x0316af to 0x006aa2 (0x2ac0c bytes) 000057 1787379277.482722 write 0x00d5ee thru 0x014633 (0x7046 bytes) truncating to largest ever: 0x3acf8 000059 1787379277.488778 trunc from 0x014634 to 0x03acf7 (0x266c4 bytes) 000060 1787379277.532917 read 0x021c5e thru 0x031267 (0xf60a bytes) 000064 1787379277.572823 read 0x02732d thru 0x02826f (0xf43 bytes) 000065 1787379277.578149 mapwrite 0x012fbd thru 0x01e7a1 (0xb7e5 bytes) 000066 1787379277.652064 mapread 0x030a09 thru 0x031784 (0xd7c bytes) 000067 1787379277.653618 read 0x0105db thru 0x01e74c (0xe172 bytes) 000068 1787379277.654785 mapread 0x0200c6 thru 0x02a29e (0xa1d9 bytes) 000069 1787379277.660520 MIRROR_EXTEND to 2 mirrors 000072 1787379278.439072 trunc from 0x03acf8 to 0x0287d1 (0x12526 bytes) 000074 1787379278.538565 read 0x004ce7 thru 0x00e526 (0x9840 bytes) 000076 1787379278.551084 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000077 1787379278.682661 trunc from 0x0287d2 to 0x0284e0 (0x2f1 bytes) 000080 1787379278.694931 mapwrite 0x00328b thru 0x00ed49 (0xbabf bytes) 000081 1787379278.781576 trunc from 0x0284e1 to 0x025c65 (0x287b bytes) 000082 1787379278.807191 mapread 0x02531f thru 0x025c65 (0x947 bytes) 000083 1787379278.824594 mapwrite 0x00d276 thru 0x01b14e (0xded9 bytes) 000085 1787379278.952025 write 0x039571 thru 0x03ffff (0x6a8f bytes) 000086 1787379278.954574 mapwrite 0x01c2ea thru 0x0258ea (0x9601 bytes) 000087 1787379279.055895 write 0x020fbc thru 0x02cefc (0xbf41 bytes) 000089 1787379279.067144 write 0x035c96 thru 0x03fdd6 (0xa141 bytes) 000091 1787379279.073534 write 0x00e132 thru 0x01139d (0x326c bytes) 000092 1787379279.083872 write 0x01737e thru 0x027202 (0xfe85 bytes) 000093 1787379279.086186 mapwrite 0x0020bc thru 0x00cb30 (0xaa75 bytes) 000096 1787379279.181607 mapwrite 0x036322 thru 0x03ffff (0x9cde bytes) 000097 1787379279.285998 read 0x018509 thru 0x01d224 (0x4d1c bytes) 000101 1787379279.294884 write 0x00d39a thru 0x00e1da (0xe41 bytes) 000103 1787379279.303474 mapwrite 0x028df6 thru 0x033b04 (0xad0f bytes) 000105 1787379279.457586 MIRROR_EXTEND to 2 mirrors 000107 1787379280.242848 write 0x01b27b thru 0x021712 (0x6498 bytes) 000108 1787379280.324041 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x66f58e9d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd15465cd Mirror 3: 0xd15465cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000109 1787379281.170042 mapread 0x03090a thru 0x030b70 (0x267 bytes) 000111 1787379281.178096 trunc from 0x040000 to 0x0114fc (0x2eb03 bytes) 000112 1787379281.266692 MIRROR_RESYNC 3 mirrors 000113 1787379282.135565 trunc from 0x0114fd to 0x019f55 (0x8a59 bytes) 000114 1787379282.291548 mapwrite 0x00918e thru 0x00920e (0x81 bytes) 000115 1787379282.397349 write 0x00341d thru 0x005664 (0x2248 bytes) 000117 1787379282.413061 write 0x01cee9 thru 0x029250 (0xc368 bytes) 000118 1787379282.416500 trunc from 0x029251 to 0x00b1e3 (0x1e06d bytes) 000119 1787379282.444854 write 0x01616e thru 0x0215a2 (0xb435 bytes) 000122 1787379282.447530 read 0x010c3f thru 0x01358a (0x294c bytes) 000125 1787379282.456855 read 0x0147c8 thru 0x017473 (0x2cac bytes) 000126 1787379282.471012 mapwrite 0x01a63b thru 0x01db15 (0x34db bytes) 000128 1787379282.536561 mapread 0x01c02d thru 0x0215a2 (0x5576 bytes) 000129 1787379282.545122 MIRROR_RESYNC 3 mirrors 000130 1787379283.289801 write 0x03dfdb thru 0x03ffff (0x2025 bytes) 000131 1787379283.350631 read 0x0030b9 thru 0x00ad4e (0x7c96 bytes) 000132 1787379283.364765 read 0x00c77c thru 0x01c4c8 (0xfd4d bytes) 000134 1787379283.387752 mapwrite 0x03657e thru 0x03e21e (0x7ca1 bytes) 000136 1787379283.441812 read 0x032414 thru 0x03fcb0 (0xd89d bytes) 000137 1787379283.443254 write 0x037ff4 thru 0x03ffff (0x800c bytes) 000138 1787379283.449257 MIRROR_RESYNC 3 mirrors 000139 1787379284.487898 read 0x006d27 thru 0x00f79b (0x8a75 bytes) 000141 1787379284.502545 mapread 0x020f1f thru 0x026f78 (0x605a bytes) 000142 1787379284.531012 write 0x00904f thru 0x014ad3 (0xba85 bytes) 000143 1787379284.624781 mapread 0x032c81 thru 0x036a25 (0x3da5 bytes) 000145 1787379284.642829 mapread 0x035b75 thru 0x03af28 (0x53b4 bytes) 000146 1787379284.643779 mapwrite 0x031be8 thru 0x039296 (0x76af bytes) 000147 1787379284.728777 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc909030d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc909030d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000148 1787379285.014605 read 0x02f3f6 thru 0x03ec12 (0xf81d bytes) 000149 1787379285.026070 read 0x012084 thru 0x01cf57 (0xaed4 bytes) 000150 1787379285.036103 write 0x01bbcd thru 0x0201d0 (0x4604 bytes) 000153 1787379285.039371 trunc from 0x040000 to 0x02c2a7 (0x13d58 bytes) 000154 1787379285.058690 write 0x021f33 thru 0x028735 (0x6803 bytes) 000157 1787379285.062483 mapread 0x021a2d thru 0x025496 (0x3a6a bytes) 000158 1787379285.063909 mapread 0x015dc7 thru 0x023278 (0xd4b2 bytes) 000159 1787379285.066007 read 0x00d76b thru 0x01826d (0xab03 bytes) 000161 1787379285.072016 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c2a8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c2a8): Mirror 2: 0x61de881f (stale) CRC-32 checksum value for chunk [0, 0x2c2a8): Mirror 1: 0xaa261cfa Mirror 3: 0xaa261cfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c2a8: skipped 000162 1787379285.770842 read 0x001600 thru 0x009103 (0x7b04 bytes) 000166 1787379285.782561 read 0x014b14 thru 0x02175a (0xcc47 bytes) 000168 1787379285.791895 write 0x017c13 thru 0x01d677 (0x5a65 bytes) 000169 1787379285.869706 write 0x01c733 thru 0x02a3c3 (0xdc91 bytes) 000170 1787379285.877781 mapread 0x013889 thru 0x018c67 (0x53df bytes) 000171 1787379285.904960 mapread 0x026f7e thru 0x02c2a7 (0x532a bytes) 000174 1787379285.925466 read 0x010cf5 thru 0x01f399 (0xe6a5 bytes) 000175 1787379285.941580 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c2a8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c2a8): Mirror 2: 0x61de881f (stale) Mirror 3: 0xaa261cfa (stale) CRC-32 checksum value for chunk [0, 0x2c2a8): Mirror 1: 0x8883ca92 Mirror 4: 0x8883ca92 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c2a8: skipped 000176 1787379287.174569 mapwrite 0x0100be thru 0x0110e0 (0x1023 bytes) 000178 1787379287.342366 mapread 0x008265 thru 0x00a0eb (0x1e87 bytes) 000181 1787379287.354014 trunc from 0x02c2a8 to 0x00825d (0x2404a bytes) 000185 1787379287.384446 read 0x0026b9 thru 0x00825d (0x5ba5 bytes) 000186 1787379287.385595 mapread 0x004ae7 thru 0x00825d (0x3777 bytes) 000187 1787379287.387922 mapread 0x003d7f thru 0x00825d (0x44df bytes) 000188 1787379287.389955 mapread 0x0022b5 thru 0x00675d (0x44a9 bytes) 000190 1787379287.393071 mapwrite 0x024b9b thru 0x025c93 (0x10f9 bytes) 000191 1787379287.461681 write 0x00ddf7 thru 0x011c59 (0x3e63 bytes) 000192 1787379287.477940 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25c94): Mirror 3: 0x310de1a6 (stale) Mirror 4: 0x93b2109 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25c94): Mirror 3: 0x310de1a6 (stale) Mirror 4: 0x93b2109 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25c94: skipped 000193 1787379287.829506 mapread 0x00307f thru 0x003329 (0x2ab bytes) 000194 1787379287.838225 trunc from 0x025c94 to 0x00aa74 (0x1b21f bytes) 000195 1787379287.863936 mapread 0x00a1d7 thru 0x00aa74 (0x89e bytes) 000196 1787379287.872733 mapwrite 0x008e63 thru 0x013616 (0xa7b4 bytes) 000198 1787379287.952605 read 0x0048c8 thru 0x013616 (0xed4f bytes) 000200 1787379287.956486 mapwrite 0x024cf1 thru 0x02d3fb (0x870b bytes) 000201 1787379288.041249 write 0x00cdd6 thru 0x01bdf6 (0xf021 bytes) 000202 1787379288.049236 mapread 0x0179bc thru 0x019fd0 (0x2615 bytes) 000203 1787379288.050648 mapread 0x00dd01 thru 0x01b396 (0xd696 bytes) 000204 1787379288.056934 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d3fc): Mirror 4: 0x49677903 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d3fc): Mirror 4: 0x49677903 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d3fc: skipped 000205 1787379288.434888 write 0x000660 thru 0x00beb8 (0xb859 bytes) 000206 1787379288.448479 trunc from 0x02d3fc to 0x032e2d (0x5a32 bytes) 000207 1787379288.469700 MIRROR_RESYNC 2 mirrors 000208 1787379289.547064 read 0x0302a2 thru 0x032e2d (0x2b8c bytes) 000210 1787379289.586791 MIRROR_EXTEND to 3 mirrors 000211 1787379290.470453 read 0x017a2f thru 0x0255bc (0xdb8e bytes) 000212 1787379290.491912 MIRROR_RESYNC 3 mirrors 000213 1787379290.902990 mapread 0x02306a thru 0x030808 (0xd79f bytes) 000214 1787379290.929276 trunc from 0x032e2e to 0x00c559 (0x268d4 bytes) 000215 1787379290.995050 trunc from 0x00c55a to 0x00e275 (0x1d1c bytes) 000219 1787379291.017636 write 0x031520 thru 0x03f0af (0xdb90 bytes) 000221 1787379291.022238 mapread 0x03ee7c thru 0x03f0af (0x234 bytes) 000223 1787379291.024594 write 0x00ae23 thru 0x00b7b6 (0x994 bytes) 000224 1787379291.033274 write 0x035e51 thru 0x03868e (0x283e bytes) 000225 1787379291.034698 trunc from 0x03f0b0 to 0x021a4e (0x1d661 bytes) 000226 1787379291.055159 trunc from 0x021a4f to 0x027a99 (0x604b bytes) 000227 1787379291.075910 mapread 0x0254da thru 0x027a99 (0x25c0 bytes) 000228 1787379291.090013 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27a9a): Mirror 4: 0x12dbabd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a9a): Mirror 4: 0x12dbabd0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a9a: skipped 000229 1787379291.429893 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27a9a): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a9a): Mirror 4: 0x12dbabd0 (stale) CRC-32 checksum value for chunk [0, 0x27a9a): Mirror 1: 0x80d3c1bd Mirror 5: 0x80d3c1bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a9a: skipped 000230 1787379292.357644 trunc from 0x027a9a to 0x001ee2 (0x25bb7 bytes) 000231 1787379292.460687 MIRROR_RESYNC 3 mirrors 000234 1787379293.474435 write 0x0144cf thru 0x016112 (0x1c44 bytes) 000238 1787379293.617676 mapread 0x00becb thru 0x015e18 (0x9f4e bytes) 000240 1787379293.662072 write 0x012392 thru 0x013e7b (0x1aea bytes) 000244 1787379293.673902 read 0x015121 thru 0x016112 (0xff2 bytes) 000246 1787379293.677452 mapread 0x00f8d5 thru 0x0158b3 (0x5fdf bytes) 000248 1787379293.684959 trunc from 0x016113 to 0x03349e (0x1d38c bytes) 000249 1787379293.725915 read 0x01d15d thru 0x025df2 (0x8c96 bytes) 000252 1787379293.765019 mapwrite 0x01ff38 thru 0x0276bc (0x7785 bytes) 000254 1787379293.922957 write 0x029357 thru 0x035e54 (0xcafe bytes) 000255 1787379293.946627 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35e55): Mirror 5: 0xcda7a08d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35e55): Mirror 5: 0xcda7a08d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35e55: skipped 000257 1787379294.405529 read 0x00c2fc thru 0x00e83e (0x2543 bytes) 000258 1787379294.410936 trunc from 0x035e55 to 0x0309a2 (0x54b2 bytes) 000260 1787379294.427636 read 0x02aa2b thru 0x0309a2 (0x5f78 bytes) 000262 1787379294.441621 mapwrite 0x0231ce thru 0x0292f9 (0x612c bytes) 000263 1787379294.499516 trunc from 0x0309a3 to 0x004134 (0x2c86e bytes) 000264 1787379294.522205 write 0x030fc7 thru 0x03e9c9 (0xda03 bytes) 000265 1787379294.531487 MIRROR_RESYNC 2 mirrors 000266 1787379295.271002 mapread 0x0360ec thru 0x03e9c9 (0x88de bytes) 000267 1787379295.300492 mapread 0x03b647 thru 0x03e9c9 (0x3383 bytes) 000268 1787379295.302103 read 0x03252c thru 0x0391fc (0x6cd1 bytes) 000270 1787379295.307816 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000272 1787379295.488459 read 0x010a9d thru 0x01af65 (0xa4c9 bytes) 000275 1787379295.508628 mapread 0x02f6e2 thru 0x0338f5 (0x4214 bytes) 000282 1787379295.547197 read 0x005000 thru 0x010733 (0xb734 bytes) 000285 1787379295.566626 mapwrite 0x016849 thru 0x020c3f (0xa3f7 bytes) 000286 1787379295.702733 mapread 0x02f4e6 thru 0x031fb2 (0x2acd bytes) 000288 1787379295.743627 write 0x01d150 thru 0x02a084 (0xcf35 bytes) 000289 1787379295.762734 mapread 0x01fc42 thru 0x02b18e (0xb54d bytes) 000290 1787379295.771791 mapread 0x014749 thru 0x014a97 (0x34f bytes) 000292 1787379295.784066 mapwrite 0x029132 thru 0x02af12 (0x1de1 bytes) 000293 1787379295.864197 mapread 0x036cf9 thru 0x03e9c9 (0x7cd1 bytes) 000297 1787379295.998143 MIRROR_EXTEND to 2 mirrors 000299 1787379297.330522 read 0x00ac47 thru 0x0194c1 (0xe87b bytes) 000301 1787379297.399453 trunc from 0x03e9ca to 0x01f025 (0x1f9a4 bytes) 000302 1787379297.607849 mapwrite 0x03efb9 thru 0x03f268 (0x2b0 bytes) 000303 1787379297.753462 write 0x03bdbe thru 0x03ffff (0x4242 bytes) 000305 1787379297.782761 trunc from 0x040000 to 0x00e31f (0x31ce0 bytes) truncating to largest ever: 0x3d61d 000306 1787379297.811544 trunc from 0x00e320 to 0x03d61c (0x2f2fd bytes) 000307 1787379297.847601 trunc from 0x03d61d to 0x037486 (0x6196 bytes) 000308 1787379297.870927 read 0x017d50 thru 0x01d0c8 (0x5379 bytes) 000311 1787379297.895721 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000315 1787379298.098097 trunc from 0x037487 to 0x024689 (0x12dfd bytes) 000316 1787379298.132446 read 0x022cdb thru 0x024689 (0x19af bytes) 000318 1787379298.142066 trunc from 0x02468a to 0x0209ed (0x3c9c bytes) 000321 1787379298.166957 read 0x019e2c thru 0x0209ed (0x6bc2 bytes) 000322 1787379298.190924 mapread 0x00d696 thru 0x016047 (0x89b2 bytes) 000325 1787379298.196979 read 0x0199c3 thru 0x0209ed (0x702b bytes) 000327 1787379298.199526 write 0x00f7b6 thru 0x017697 (0x7ee2 bytes) 000328 1787379298.205398 mapwrite 0x01cf6c thru 0x02c156 (0xf1eb bytes) 000329 1787379298.315888 MIRROR_EXTEND to 2 mirrors 000331 1787379299.404847 trunc from 0x02c157 to 0x03909c (0xcf46 bytes) 000332 1787379299.668738 read 0x022fb2 thru 0x030063 (0xd0b2 bytes) 000333 1787379299.695247 mapread 0x03445f thru 0x03722e (0x2dd0 bytes) 000334 1787379299.743356 write 0x000946 thru 0x002565 (0x1c20 bytes) 000336 1787379299.772751 mapread 0x002dc5 thru 0x0098de (0x6b1a bytes) 000337 1787379299.804231 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3909d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3909d): Mirror 2: 0xdfcfca35 (stale) CRC-32 checksum value for chunk [0, 0x3909d): Mirror 1: 0xa9efa20c Mirror 3: 0xa9efa20c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3909d: skipped 000338 1787379301.324842 trunc from 0x03909d to 0x02ec90 (0xa40c bytes) 000339 1787379301.540790 write 0x033583 thru 0x034d8d (0x180b bytes) 000340 1787379301.543078 mapread 0x018108 thru 0x01cc88 (0x4b81 bytes) 000341 1787379301.600998 write 0x017712 thru 0x022856 (0xb145 bytes) 000342 1787379301.622208 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34d8e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d8e): Mirror 2: 0x86c67a0f (stale) Mirror 3: 0x6dd747d3 (stale) CRC-32 checksum value for chunk [0, 0x34d8e): Mirror 1: 0x95b28a36 Mirror 4: 0x95b28a36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d8e: skipped 000343 1787379303.231580 mapwrite 0x01de3b thru 0x02440f (0x65d5 bytes) 000346 1787379303.597087 write 0x016f7e thru 0x025e59 (0xeedc bytes) 000347 1787379303.616991 read 0x023034 thru 0x031559 (0xe526 bytes) 000349 1787379303.652704 mapread 0x0051a5 thru 0x014e6b (0xfcc7 bytes) 000350 1787379303.696248 mapread 0x021284 thru 0x0301f9 (0xef76 bytes) 000355 1787379303.720603 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34d8e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34d8e): Mirror 2: 0x86c67a0f (stale) Mirror 3: 0x6dd747d3 (stale) Mirror 4: 0x95b28a36 (stale) CRC-32 checksum value for chunk [0, 0x34d8e): Mirror 1: 0x590ddde5 Mirror 5: 0x590ddde5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34d8e: skipped 000356 1787379305.880089 trunc from 0x034d8e to 0x016b1b (0x1e272 bytes) 000358 1787379306.136922 mapread 0x00569e thru 0x00e40e (0x8d71 bytes) 000360 1787379306.204196 trunc from 0x016b1c to 0x00cbd9 (0x9f42 bytes) 000362 1787379306.239136 write 0x005ab9 thru 0x00c676 (0x6bbe bytes) 000363 1787379306.246222 trunc from 0x00cbda to 0x01a0be (0xd4e5 bytes) 000367 1787379306.313176 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a0bf): Mirror 3: 0xfc5e543e (stale) Mirror 4: 0xac18f675 (stale) Mirror 5: 0xd12bfab4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a0bf): Mirror 3: 0xfc5e543e (stale) Mirror 4: 0xac18f675 (stale) Mirror 5: 0xd12bfab4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a0bf: skipped 000368 1787379307.083112 trunc from 0x01a0bf to 0x010939 (0x9785 bytes) 000370 1787379307.130865 write 0x029e47 thru 0x038494 (0xe64e bytes) 000372 1787379307.147713 mapwrite 0x027b7f thru 0x02a7cf (0x2c51 bytes) 000379 1787379307.445114 mapwrite 0x017073 thru 0x01873e (0x16cc bytes) 000380 1787379307.648383 mapread 0x037ec5 thru 0x038494 (0x5d0 bytes) 000382 1787379307.669448 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38495): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38495): Mirror 3: 0x8f6dda05 (stale) Mirror 4: 0x46ea6e29 (stale) Mirror 5: 0x7de907d3 (stale) CRC-32 checksum value for chunk [0, 0x38495): Mirror 1: 0x5c64ad45 Mirror 6: 0x5c64ad45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38495: skipped 000384 1787379308.967800 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38495): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38495): Mirror 3: 0x8f6dda05 (stale) Mirror 4: 0x46ea6e29 (stale) Mirror 5: 0x7de907d3 (stale) CRC-32 checksum value for chunk [0, 0x38495): Mirror 1: 0x5c64ad45 Mirror 6: 0x5c64ad45 Mirror 7: 0x5c64ad45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38495: skipped 000388 1787379310.010168 trunc from 0x038495 to 0x0255fe (0x12e96 bytes) 000389 1787379310.104657 write 0x027616 thru 0x03434c (0xcd37 bytes) 000390 1787379310.107918 trunc from 0x03434d to 0x01d7a3 (0x16ba9 bytes) 000391 1787379310.134683 mapwrite 0x0255ce thru 0x02dd57 (0x878a bytes) 000393 1787379310.281966 MIRROR_RESYNC 6 mirrors 000396 1787379311.312888 mapwrite 0x03110b thru 0x03b271 (0xa167 bytes) 000397 1787379311.580550 mapwrite 0x0112dc thru 0x020d53 (0xfa78 bytes) 000398 1787379311.687775 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b272): Mirror 3: 0x1db46b16 (stale) Mirror 4: 0x1db46b16 (stale) Mirror 5: 0x1db46b16 (stale) Mirror 7: 0x1db46b16 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b272): Mirror 3: 0x1db46b16 (stale) Mirror 4: 0x1db46b16 (stale) Mirror 5: 0x1db46b16 (stale) Mirror 7: 0x1db46b16 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b272: skipped 000399 1787379312.279884 write 0x02c416 thru 0x02d1ef (0xdda bytes) 000400 1787379312.322916 mapread 0x02724b thru 0x02f3ee (0x81a4 bytes) 000401 1787379312.367915 trunc from 0x03b272 to 0x01fb32 (0x1b73f bytes) 000404 1787379312.395420 mapwrite 0x011fce thru 0x014f30 (0x2f63 bytes) 000405 1787379312.646153 write 0x036332 thru 0x03ffff (0x9cce bytes) 000407 1787379312.662632 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xba39e2bc (stale) Mirror 4: 0xba39e2bc (stale) Mirror 5: 0xba39e2bc (stale) Mirror 7: 0xba39e2bc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x69671e78 Mirror 8: 0x69671e78 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000410 1787379313.863032 mapread 0x0178b1 thru 0x023ce0 (0xc430 bytes) 000411 1787379313.903101 read 0x015c29 thru 0x020997 (0xad6f bytes) 000412 1787379313.910540 mapwrite 0x01ad58 thru 0x020193 (0x543c bytes) 000414 1787379314.081384 write 0x011a64 thru 0x01b7bd (0x9d5a bytes) 000415 1787379314.092108 mapread 0x005ac0 thru 0x00e133 (0x8674 bytes) 000419 1787379314.111837 trunc from 0x040000 to 0x00853c (0x37ac3 bytes) 000424 1787379314.168136 read 0x005652 thru 0x00853c (0x2eeb bytes) 000425 1787379314.170906 mapwrite 0x005fdc thru 0x00f206 (0x922b bytes) 000426 1787379314.284776 mapread 0x00707c thru 0x007142 (0xc7 bytes) 000427 1787379314.285979 mapread 0x0032ab thru 0x00535b (0x20b1 bytes) 000428 1787379314.287921 write 0x019523 thru 0x01fc4a (0x6728 bytes) 000429 1787379314.291873 trunc from 0x01fc4b to 0x01e7af (0x149b bytes) 000431 1787379314.314737 mapread 0x0060f8 thru 0x009b9a (0x3aa3 bytes) 000432 1787379314.327176 MIRROR_RESYNC 6 mirrors 000433 1787379315.844887 read 0x00bcbe thru 0x01b9c2 (0xfd05 bytes) 000434 1787379315.859105 mapwrite 0x004da1 thru 0x01005b (0xb2bb bytes) truncating to largest ever: 0x3f7ae 000437 1787379316.137391 trunc from 0x01e7b0 to 0x03f7ad (0x20ffe bytes) 000438 1787379316.196432 mapwrite 0x004563 thru 0x004bd2 (0x670 bytes) 000439 1787379316.289113 write 0x01a5dd thru 0x02860d (0xe031 bytes) 000440 1787379316.332867 mapread 0x01c3f1 thru 0x01dc0a (0x181a bytes) 000441 1787379316.342138 trunc from 0x03f7ae to 0x00389f (0x3bf0e bytes) 000442 1787379316.402046 read 0x00318d thru 0x00389f (0x713 bytes) 000443 1787379316.403035 read 0x001cbb thru 0x00389f (0x1be5 bytes) 000445 1787379316.406826 trunc from 0x0038a0 to 0x02c4e1 (0x28c42 bytes) 000446 1787379316.445198 trunc from 0x02c4e2 to 0x0074a7 (0x2503a bytes) 000447 1787379316.468796 write 0x007762 thru 0x007cbf (0x55e bytes) 000448 1787379316.482366 mapwrite 0x01c260 thru 0x029b4c (0xd8ed bytes) 000450 1787379316.672082 write 0x0171bf thru 0x020640 (0x9482 bytes) 000453 1787379316.682911 write 0x032e35 thru 0x033240 (0x40c bytes) 000454 1787379316.685660 mapread 0x014a6b thru 0x021847 (0xcddd bytes) 000456 1787379316.688545 read 0x003bc5 thru 0x004c40 (0x107c bytes) 000457 1787379316.700332 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33241): Mirror 3: 0xb4fb14fe (stale) Mirror 4: 0xb4fb14fe (stale) Mirror 5: 0xb4fb14fe (stale) Mirror 7: 0xb4fb14fe (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33241): Mirror 3: 0xb4fb14fe (stale) Mirror 4: 0xb4fb14fe (stale) Mirror 5: 0xb4fb14fe (stale) Mirror 7: 0xb4fb14fe (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33241: skipped 000458 1787379317.233088 mapwrite 0x0020d4 thru 0x00c977 (0xa8a4 bytes) 000460 1787379317.375054 MIRROR_RESYNC 5 mirrors 000461 1787379318.392810 mapwrite 0x022a40 thru 0x0268d5 (0x3e96 bytes) 000465 1787379318.608479 MIRROR_RESYNC 5 mirrors 000468 1787379319.501163 mapread 0x009c2d thru 0x018d30 (0xf104 bytes) 000469 1787379319.533001 trunc from 0x033241 to 0x02dd4a (0x54f6 bytes) 000470 1787379319.603985 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd4b): Mirror 3: 0xa4c4592b (stale) Mirror 5: 0xa4c4592b (stale) Mirror 7: 0xa4c4592b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd4b): Mirror 3: 0xa4c4592b (stale) Mirror 5: 0xa4c4592b (stale) Mirror 7: 0xa4c4592b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd4b: skipped 000472 1787379320.068671 mapread 0x012a1c thru 0x01db0b (0xb0f0 bytes) 000473 1787379320.116688 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2dd4b): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd4b): Mirror 3: 0xa4c4592b (stale) Mirror 5: 0xa4c4592b (stale) Mirror 7: 0xa4c4592b (stale) CRC-32 checksum value for chunk [0, 0x2dd4b): Mirror 1: 0xa4c4592b Mirror 8: 0xa4c4592b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd4b: skipped 000475 1787379321.238063 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd4b): Mirror 5: 0xa4c4592b (stale) Mirror 7: 0xa4c4592b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd4b): Mirror 5: 0xa4c4592b (stale) Mirror 7: 0xa4c4592b (stale) CRC-32 checksum value for chunk [0, 0x2dd4b): Mirror 1: 0xa4c4592b Mirror 8: 0xa4c4592b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd4b: skipped 000477 1787379321.861666 read 0x01dcce thru 0x01fdf5 (0x2128 bytes) 000478 1787379321.887712 write 0x02999b thru 0x034560 (0xabc6 bytes) 000479 1787379321.979052 mapread 0x031cc9 thru 0x034560 (0x2898 bytes) 000480 1787379321.980319 mapwrite 0x008e4a thru 0x0168ec (0xdaa3 bytes) 000481 1787379322.077078 MIRROR_RESYNC 4 mirrors 000483 1787379323.305386 mapwrite 0x028356 thru 0x02ab67 (0x2812 bytes) 000488 1787379323.472660 trunc from 0x034561 to 0x0373f8 (0x2e98 bytes) 000489 1787379323.489693 mapread 0x022e36 thru 0x025e37 (0x3002 bytes) 000490 1787379323.491072 trunc from 0x0373f9 to 0x0353f7 (0x2001 bytes) 000492 1787379323.506094 mapread 0x024e28 thru 0x02ab89 (0x5d62 bytes) 000493 1787379323.507347 write 0x000188 thru 0x00b83c (0xb6b5 bytes) 000499 1787379323.526142 write 0x008a49 thru 0x00e1bb (0x5773 bytes) 000501 1787379323.532743 trunc from 0x0353f8 to 0x018d16 (0x1c6e1 bytes) 000505 1787379323.549958 mapread 0x008a8a thru 0x00bdb8 (0x332f bytes) 000507 1787379323.552893 mapread 0x016774 thru 0x018d16 (0x25a3 bytes) 000508 1787379323.567743 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18d17): Mirror 5: 0x16b8543f (stale) Mirror 8: 0x16b8543f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18d17): Mirror 5: 0x16b8543f (stale) Mirror 8: 0x16b8543f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18d17: skipped 000509 1787379323.888918 MIRROR_RESYNC 3 mirrors 000510 1787379324.535249 mapwrite 0x02bf47 thru 0x035e7a (0x9f34 bytes) 000511 1787379324.626949 mapread 0x027ac4 thru 0x0312ee (0x982b bytes) 000514 1787379324.631238 trunc from 0x035e7b to 0x0333dc (0x2a9e bytes) 000515 1787379324.674268 mapwrite 0x03789d thru 0x0394c3 (0x1c27 bytes) 000517 1787379324.783232 MIRROR_RESYNC 3 mirrors 000519 1787379325.659461 mapread 0x016fda thru 0x023b85 (0xcbac bytes) 000520 1787379325.751608 mapwrite 0x0065c7 thru 0x011aa4 (0xb4de bytes) 000522 1787379326.003063 write 0x02cb5c thru 0x031ea8 (0x534d bytes) 000523 1787379326.033629 trunc from 0x0394c4 to 0x01c03a (0x1d489 bytes) 000524 1787379326.060661 mapread 0x019dbc thru 0x01c03a (0x227f bytes) 000525 1787379326.076380 mapread 0x008bee thru 0x0174ab (0xe8be bytes) 000526 1787379326.080848 mapwrite 0x01fc6d thru 0x02d541 (0xd8d5 bytes) 000528 1787379326.328108 read 0x024b5c thru 0x02d541 (0x89e6 bytes) 000530 1787379326.332031 mapread 0x00e5eb thru 0x01cf6c (0xe982 bytes) 000531 1787379326.333452 write 0x0262e2 thru 0x02ede4 (0x8b03 bytes) 000532 1787379326.339322 mapread 0x006ef4 thru 0x00956c (0x2679 bytes) 000534 1787379326.352174 MIRROR_RESYNC 3 mirrors 000536 1787379327.186665 MIRROR_RESYNC 3 mirrors 000537 1787379327.837725 MIRROR_EXTEND to 4 mirrors 000538 1787379328.689628 trunc from 0x02ede5 to 0x012e0f (0x1bfd5 bytes) 000540 1787379328.812146 mapread 0x009522 thru 0x012e0f (0x98ee bytes) 000541 1787379328.840581 write 0x01b34c thru 0x01db7c (0x2831 bytes) 000542 1787379328.863752 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1db7d): Mirror 8: 0x29d8cdc6 (stale) Mirror 9: 0x29d8cdc6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1db7d): Mirror 8: 0x29d8cdc6 (stale) Mirror 9: 0x29d8cdc6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1db7d: skipped 000546 1787379329.274588 mapread 0x00d4cd thru 0x01af29 (0xda5d bytes) 000547 1787379329.301881 read 0x0108aa thru 0x01d079 (0xc7d0 bytes) 000549 1787379329.308258 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1db7d): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1db7d): Mirror 8: 0x29d8cdc6 (stale) Mirror 9: 0x29d8cdc6 (stale) CRC-32 checksum value for chunk [0, 0x1db7d): Mirror 1: 0x86c61e Mirror 10: 0x86c61e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1db7d: skipped 000550 1787379330.133846 trunc from 0x01db7d to 0x02e73d (0x10bc1 bytes) 000551 1787379330.191998 mapwrite 0x011fe9 thru 0x018ba6 (0x6bbe bytes) 000556 1787379330.270760 mapwrite 0x01d647 thru 0x01f94c (0x2306 bytes) 000557 1787379330.332650 read 0x008ef2 thru 0x0138f9 (0xaa08 bytes) 000559 1787379330.346758 mapread 0x01b9a4 thru 0x01e9ec (0x3049 bytes) 000562 1787379330.351612 trunc from 0x02e73e to 0x03d34c (0xec0f bytes) 000565 1787379330.386487 write 0x020e19 thru 0x022943 (0x1b2b bytes) 000566 1787379330.395755 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d34d): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d34d): Mirror 8: 0x28b9c39d (stale) Mirror 9: 0x28b9c39d (stale) Mirror 10: 0x21a3a0cf (stale) CRC-32 checksum value for chunk [0, 0x3d34d): Mirror 1: 0xb0dcf043 Mirror 11: 0xb0dcf043 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d34d: skipped 000567 1787379331.734451 read 0x0238f0 thru 0x032650 (0xed61 bytes) 000568 1787379331.754871 mapwrite 0x03333c thru 0x03ffff (0xccc4 bytes) 000570 1787379331.987690 trunc from 0x040000 to 0x03f33b (0xcc4 bytes) 000572 1787379332.011030 read 0x016e24 thru 0x01ab67 (0x3d44 bytes) 000573 1787379332.045257 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f33c): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f33c): Mirror 8: 0x7e9f59cf (stale) Mirror 9: 0x7e9f59cf (stale) Mirror 10: 0xe12e5825 (stale) Mirror 11: 0x18a475df (stale) CRC-32 checksum value for chunk [0, 0x3f33c): Mirror 1: 0xe6e7535e Mirror 12: 0xe6e7535e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f33c: skipped 000574 1787379332.995924 read 0x007daa thru 0x00d804 (0x5a5b bytes) 000577 1787379333.031663 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f33c): Mirror 8: 0x7e9f59cf (stale) Mirror 10: 0xe12e5825 (stale) Mirror 11: 0x18a475df (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f33c): Mirror 8: 0x7e9f59cf (stale) Mirror 10: 0xe12e5825 (stale) Mirror 11: 0x18a475df (stale) CRC-32 checksum value for chunk [0, 0x3f33c): Mirror 1: 0xe6e7535e Mirror 12: 0xe6e7535e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f33c: skipped 000579 1787379333.881053 read 0x00e439 thru 0x01cb28 (0xe6f0 bytes) 000580 1787379333.927463 mapread 0x012596 thru 0x019f07 (0x7972 bytes) 000582 1787379334.036807 mapwrite 0x032c0d thru 0x0386d3 (0x5ac7 bytes) 000583 1787379334.342953 mapwrite 0x0067ee thru 0x0144f6 (0xdd09 bytes) 000585 1787379334.592460 mapread 0x01794f thru 0x01a45f (0x2b11 bytes) 000586 1787379334.629907 mapread 0x02ba9f thru 0x038863 (0xcdc5 bytes) 000587 1787379334.652251 trunc from 0x03f33c to 0x03a8ec (0x4a4f bytes) 000589 1787379334.694770 trunc from 0x03a8ed to 0x032555 (0x8397 bytes) 000590 1787379334.715532 read 0x012cf7 thru 0x01d7c6 (0xaad0 bytes) 000591 1787379334.716274 mapread 0x02ed6e thru 0x032555 (0x37e8 bytes) 000593 1787379334.729075 MIRROR_RESYNC 5 mirrors 000594 1787379336.159419 read 0x00b35c thru 0x01125a (0x5eff bytes) 000595 1787379336.170840 trunc from 0x032556 to 0x033d40 (0x17eb bytes) 000597 1787379336.265012 mapread 0x00f770 thru 0x013873 (0x4104 bytes) 000598 1787379336.289642 write 0x01c20a thru 0x02ad57 (0xeb4e bytes) 000601 1787379336.301458 mapread 0x010aa2 thru 0x01b314 (0xa873 bytes) 000602 1787379336.307658 mapwrite 0x02e465 thru 0x0394e6 (0xb082 bytes) 000603 1787379336.393725 write 0x01b934 thru 0x022dce (0x749b bytes) 000604 1787379336.400396 MIRROR_RESYNC 5 mirrors 000606 1787379337.234492 read 0x025cd6 thru 0x02a9ae (0x4cd9 bytes) 000608 1787379337.243190 trunc from 0x0394e7 to 0x03f731 (0x624b bytes) 000609 1787379337.335082 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f732): Mirror 8: 0x3c64552b (stale) Mirror 10: 0x3c64552b (stale) Mirror 11: 0x3c64552b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f732): Mirror 8: 0x3c64552b (stale) Mirror 10: 0x3c64552b (stale) Mirror 11: 0x3c64552b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f732: skipped 000610 1787379337.660483 mapwrite 0x0286a6 thru 0x02f44b (0x6da6 bytes) 000611 1787379337.727348 mapwrite 0x015193 thru 0x015b58 (0x9c6 bytes) 000612 1787379337.787874 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f732): Mirror 10: 0x3c64552b (stale) Mirror 11: 0x3c64552b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f732): Mirror 10: 0x3c64552b (stale) Mirror 11: 0x3c64552b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f732: skipped 000613 1787379338.089351 mapwrite 0x03d05f thru 0x03ffff (0x2fa1 bytes) 000614 1787379338.147842 mapread 0x01d946 thru 0x01e974 (0x102f bytes) 000615 1787379338.163609 mapwrite 0x039713 thru 0x03b173 (0x1a61 bytes) 000616 1787379338.208546 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xd7b56fa6 (stale) Mirror 11: 0xd7b56fa6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x36e44970 Mirror 12: 0x36e44970 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000617 1787379338.901113 write 0x03e325 thru 0x03ffff (0x1cdb bytes) 000620 1787379338.988161 write 0x03245b thru 0x03725b (0x4e01 bytes) 000621 1787379338.998156 trunc from 0x040000 to 0x037218 (0x8de7 bytes) 000622 1787379339.034468 trunc from 0x037219 to 0x03e152 (0x6f3a bytes) 000623 1787379339.051462 trunc from 0x03e153 to 0x01a3cd (0x23d85 bytes) 000624 1787379339.071153 write 0x01a2dd thru 0x0247f6 (0xa51a bytes) 000626 1787379339.079737 write 0x03b9cf thru 0x03ffff (0x4631 bytes) 000627 1787379339.082088 trunc from 0x040000 to 0x00d7bd (0x32842 bytes) 000629 1787379339.116037 MIRROR_RESYNC 4 mirrors 000631 1787379339.812808 read 0x00cf53 thru 0x00d7bd (0x86b bytes) 000634 1787379339.822003 write 0x021f0e thru 0x0286db (0x67ce bytes) 000637 1787379339.898247 trunc from 0x0286dc to 0x03e3fd (0x15d22 bytes) 000639 1787379339.918220 mapread 0x011810 thru 0x021722 (0xff13 bytes) 000640 1787379339.935821 write 0x03dcd1 thru 0x03f6a8 (0x19d8 bytes) 000641 1787379339.939932 trunc from 0x03f6a9 to 0x02d793 (0x11f15 bytes) 000642 1787379339.988612 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d794): Mirror 10: 0x6aacef97 (stale) Mirror 11: 0x6aacef97 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d794): Mirror 10: 0x6aacef97 (stale) Mirror 11: 0x6aacef97 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d794: skipped 000645 1787379340.306827 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d794): Mirror 11: 0x6aacef97 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d794): Mirror 11: 0x6aacef97 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d794: skipped 000648 1787379340.639758 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d794): Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d794): Mirror 11: 0x6aacef97 (stale) CRC-32 checksum value for chunk [0, 0x2d794): Mirror 1: 0xe7564e6f Mirror 12: 0xe7564e6f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d794: skipped 000649 1787379341.349992 trunc from 0x02d794 to 0x00a7bb (0x22fd8 bytes) 000650 1787379341.463076 mapread 0x001409 thru 0x00a7bb (0x93b3 bytes) 000652 1787379341.485735 mapwrite 0x02944a thru 0x033be5 (0xa79c bytes) 000653 1787379341.592826 mapread 0x024ba6 thru 0x02c8a6 (0x7d01 bytes) 000655 1787379341.597591 mapwrite 0x018751 thru 0x027b30 (0xf3e0 bytes) 000657 1787379341.669406 mapwrite 0x0388ec thru 0x03ffff (0x7714 bytes) 000658 1787379341.760262 MIRROR_RESYNC 3 mirrors 000660 1787379342.407661 trunc from 0x040000 to 0x0025c1 (0x3da3e bytes) 000663 1787379342.470395 read 0x001398 thru 0x0025c1 (0x122a bytes) 000665 1787379342.485995 MIRROR_RESYNC 3 mirrors 000668 1787379343.140499 read 0x0013d5 thru 0x0025c1 (0x11ed bytes) 000672 1787379343.150818 read 0x000e2e thru 0x0025c1 (0x1794 bytes) 000674 1787379343.159779 trunc from 0x0025c2 to 0x01972d (0x1716c bytes) 000675 1787379343.284112 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1972e): Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1972e): Mirror 11: 0x4578c64f (stale) Mirror 12: 0x4578c64f (stale) CRC-32 checksum value for chunk [0, 0x1972e): Mirror 1: 0x4578c64f Mirror 13: 0x4578c64f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1972e: skipped 000677 1787379344.119731 read 0x019611 thru 0x01972d (0x11d bytes) 000678 1787379344.154380 mapread 0x00b96d thru 0x0107bc (0x4e50 bytes) 000680 1787379344.239065 mapread 0x006b58 thru 0x01055d (0x9a06 bytes) 000683 1787379344.267266 trunc from 0x01972e to 0x009c99 (0xfa94 bytes) 000684 1787379344.321552 write 0x000632 thru 0x008519 (0x7ee8 bytes) 000685 1787379344.334235 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9c9a): Mirror 12: 0x90bbbb8d (stale) Mirror 13: 0x90bbbb8d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9c9a): Mirror 12: 0x90bbbb8d (stale) Mirror 13: 0x90bbbb8d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9c9a: skipped 000686 1787379344.595590 mapread 0x006369 thru 0x009c99 (0x3931 bytes) 000687 1787379344.613570 MIRROR_RESYNC 3 mirrors 000688 1787379345.449105 write 0x02bba9 thru 0x030887 (0x4cdf bytes) 000689 1787379345.531852 read 0x026d6d thru 0x030887 (0x9b1b bytes) 000690 1787379345.540341 write 0x013099 thru 0x020153 (0xd0bb bytes) 000691 1787379345.550734 mapread 0x01cf96 thru 0x0228c5 (0x5930 bytes) 000692 1787379345.560343 mapread 0x01a9f7 thru 0x025dd4 (0xb3de bytes) 000693 1787379345.567473 write 0x00a4db thru 0x015fa3 (0xbac9 bytes) 000694 1787379345.574135 read 0x015829 thru 0x01e735 (0x8f0d bytes) 000695 1787379345.574920 mapwrite 0x00bcda thru 0x01a23a (0xe561 bytes) 000699 1787379345.626210 mapwrite 0x01cae2 thru 0x0268e8 (0x9e07 bytes) 000700 1787379345.669955 mapwrite 0x015165 thru 0x01f7e9 (0xa685 bytes) 000702 1787379345.714648 trunc from 0x030888 to 0x02760d (0x927a bytes) 000705 1787379345.735427 write 0x00fde1 thru 0x01abc2 (0xade2 bytes) 000707 1787379345.738114 mapwrite 0x006744 thru 0x0085a9 (0x1e66 bytes) 000709 1787379345.796014 read 0x016b3f thru 0x022001 (0xb4c3 bytes) 000711 1787379345.797466 write 0x03199e thru 0x036caf (0x5312 bytes) 000714 1787379345.802627 trunc from 0x036cb0 to 0x039296 (0x25e7 bytes) 000715 1787379345.826012 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39297): Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39297): Mirror 12: 0x98aea691 (stale) Mirror 13: 0x98aea691 (stale) CRC-32 checksum value for chunk [0, 0x39297): Mirror 1: 0xcf19b747 Mirror 14: 0xcf19b747 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39297: skipped 000716 1787379346.600840 trunc from 0x039297 to 0x0396cf (0x439 bytes) 000718 1787379346.689449 mapread 0x0122ff thru 0x01c0fc (0x9dfe bytes) 000719 1787379346.712922 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x396d0): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396d0): Mirror 12: 0xbfa6d95e (stale) Mirror 13: 0xbfa6d95e (stale) Mirror 14: 0x91e43b97 (stale) CRC-32 checksum value for chunk [0, 0x396d0): Mirror 1: 0x91e43b97 Mirror 15: 0x91e43b97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396d0: skipped 000720 1787379347.687519 mapread 0x02ca8c thru 0x02e3aa (0x191f bytes) 000725 1787379347.820263 trunc from 0x0396d0 to 0x0272fd (0x123d2 bytes) 000726 1787379347.954182 trunc from 0x0272fe to 0x01ecbb (0x8642 bytes) 000728 1787379348.007803 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ecbc): Mirror 12: 0x1b30f5ba (stale) Mirror 13: 0x1b30f5ba (stale) Mirror 14: 0x1d141ebf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ecbc): Mirror 12: 0x1b30f5ba (stale) Mirror 13: 0x1b30f5ba (stale) Mirror 14: 0x1d141ebf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ecbc: skipped 000729 1787379348.373183 read 0x016fd1 thru 0x01ccb0 (0x5ce0 bytes) 000730 1787379348.384768 mapread 0x019a47 thru 0x01b003 (0x15bd bytes) 000731 1787379348.389237 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1ecbc): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ecbc): Mirror 12: 0x1b30f5ba (stale) Mirror 13: 0x1b30f5ba (stale) Mirror 14: 0x1d141ebf (stale) CRC-32 checksum value for chunk [0, 0x1ecbc): Mirror 1: 0x1d141ebf Mirror 15: 0x1d141ebf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ecbc: skipped 000732 1787379349.718864 write 0x01ac11 thru 0x01efc3 (0x43b3 bytes) 000733 1787379349.860651 mapread 0x0144d7 thru 0x017f2c (0x3a56 bytes) 000735 1787379349.888835 write 0x03ad49 thru 0x03e20e (0x34c6 bytes) 000737 1787379349.892694 trunc from 0x03e20f to 0x01d9f7 (0x20817 bytes) 000738 1787379349.915841 mapread 0x01b753 thru 0x01d9f7 (0x22a5 bytes) 000739 1787379349.916941 mapread 0x002042 thru 0x0032ee (0x12ad bytes) 000744 1787379349.932261 write 0x01dcb8 thru 0x02230d (0x4656 bytes) 000745 1787379349.934011 write 0x01e459 thru 0x029391 (0xaf39 bytes) 000746 1787379349.936250 mapread 0x0041e8 thru 0x00529f (0x10b8 bytes) 000750 1787379349.940981 trunc from 0x029392 to 0x03f105 (0x15d74 bytes) 000751 1787379349.962762 write 0x03bc67 thru 0x03ffff (0x4399 bytes) 000753 1787379349.969577 mapwrite 0x023b68 thru 0x02d9b4 (0x9e4d bytes) 000757 1787379350.046514 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7cb1add6 (stale) Mirror 13: 0x7cb1add6 (stale) Mirror 15: 0x5383f277 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7cb1add6 (stale) Mirror 13: 0x7cb1add6 (stale) Mirror 15: 0x5383f277 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000760 1787379350.422645 read 0x023912 thru 0x023cc0 (0x3af bytes) 000762 1787379350.429809 read 0x0210c8 thru 0x024d13 (0x3c4c bytes) 000764 1787379350.442802 mapread 0x00c638 thru 0x00d1ac (0xb75 bytes) 000765 1787379350.444474 read 0x0016a6 thru 0x004347 (0x2ca2 bytes) 000767 1787379350.446612 write 0x011314 thru 0x0189c8 (0x76b5 bytes) 000770 1787379350.451256 read 0x015c27 thru 0x020d0e (0xb0e8 bytes) 000772 1787379350.457813 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7cb1add6 (stale) Mirror 13: 0x7cb1add6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7cb1add6 (stale) Mirror 13: 0x7cb1add6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000773 1787379350.766981 write 0x038997 thru 0x03ffff (0x7669 bytes) 000777 1787379350.778126 write 0x0125d3 thru 0x0136d6 (0x1104 bytes) 000779 1787379350.788916 mapread 0x01a9cb thru 0x01ba74 (0x10aa bytes) 000780 1787379350.804458 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7cb1add6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x7cb1add6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000781 1787379351.183162 write 0x02cadc thru 0x03734b (0xa870 bytes) 000782 1787379351.197008 mapread 0x0140ce thru 0x021759 (0xd68c bytes) 000783 1787379351.227083 write 0x02763b thru 0x0292a4 (0x1c6a bytes) 000784 1787379351.230766 read 0x0068b7 thru 0x00be20 (0x556a bytes) 000785 1787379351.237210 read 0x01eb37 thru 0x02bdd4 (0xd29e bytes) 000787 1787379351.248416 mapread 0x0228dc thru 0x0262ca (0x39ef bytes) 000788 1787379351.249941 write 0x01dbcd thru 0x020473 (0x28a7 bytes) 000791 1787379351.255443 mapwrite 0x01b4c3 thru 0x01ee7e (0x39bc bytes) 000792 1787379351.302026 write 0x03b30a thru 0x03ffff (0x4cf6 bytes) 000794 1787379351.306655 mapwrite 0x035b65 thru 0x03ffff (0xa49b bytes) 000795 1787379351.355042 read 0x0347b3 thru 0x03fdb6 (0xb604 bytes) 000796 1787379351.361785 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000797 1787379351.509149 write 0x0331b2 thru 0x03ffff (0xce4e bytes) 000799 1787379351.524036 MIRROR_EXTEND to 2 mirrors 000801 1787379352.081365 read 0x030e86 thru 0x039434 (0x85af bytes) 000803 1787379352.099529 MIRROR_RESYNC 2 mirrors 000804 1787379352.393941 write 0x00d107 thru 0x00e398 (0x1292 bytes) 000805 1787379352.446608 mapwrite 0x035b68 thru 0x03ffff (0xa498 bytes) 000806 1787379352.508564 mapread 0x010694 thru 0x01d826 (0xd193 bytes) 000808 1787379352.521246 mapwrite 0x03e6d0 thru 0x03ffff (0x1930 bytes) 000809 1787379352.554467 read 0x00ccdf thru 0x018ee6 (0xc208 bytes) 000810 1787379352.555435 mapread 0x02185d thru 0x023872 (0x2016 bytes) 000812 1787379352.568685 mapwrite 0x00b779 thru 0x0167a3 (0xb02b bytes) 000813 1787379352.641075 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000816 1787379352.788812 write 0x002f1a thru 0x012794 (0xf87b bytes) 000817 1787379352.799168 mapread 0x006bc9 thru 0x0087fb (0x1c33 bytes) 000818 1787379352.800084 mapwrite 0x03435c thru 0x03cfd5 (0x8c7a bytes) 000819 1787379352.856895 trunc from 0x040000 to 0x0062e5 (0x39d1a bytes) 000820 1787379352.897874 mapwrite 0x03934b thru 0x03febd (0x6b73 bytes) 000821 1787379353.044003 MIRROR_EXTEND to 2 mirrors 000823 1787379353.802843 mapread 0x0026d0 thru 0x009d17 (0x7648 bytes) 000825 1787379353.842238 write 0x03b80b thru 0x03ffff (0x47f5 bytes) 000826 1787379353.918118 trunc from 0x040000 to 0x0319a7 (0xe658 bytes) 000827 1787379353.950781 trunc from 0x0319a8 to 0x03488e (0x2ee7 bytes) 000832 1787379353.970877 write 0x00c161 thru 0x01acdd (0xeb7d bytes) 000835 1787379353.987925 MIRROR_RESYNC 2 mirrors 000836 1787379354.511421 write 0x000005 thru 0x004bf6 (0x4bf2 bytes) 000837 1787379354.585140 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3488f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3488f): Mirror 2: 0xada2cd06 (stale) CRC-32 checksum value for chunk [0, 0x3488f): Mirror 1: 0xc74284b9 Mirror 3: 0xc74284b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3488f: skipped 000838 1787379355.439061 mapread 0x0176dd thru 0x01f915 (0x8239 bytes) 000840 1787379355.466248 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3488f): Mirror 2: 0xada2cd06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3488f): Mirror 2: 0xada2cd06 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3488f: skipped 000841 1787379355.702954 mapwrite 0x005c08 thru 0x010f4d (0xb346 bytes) 000842 1787379355.785333 write 0x003e37 thru 0x00ea81 (0xac4b bytes) 000844 1787379355.789893 trunc from 0x03488f to 0x035180 (0x8f2 bytes) 000845 1787379355.810008 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35181): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35181): Mirror 2: 0xd39b10cb (stale) CRC-32 checksum value for chunk [0, 0x35181): Mirror 1: 0xe86c3908 Mirror 3: 0xe86c3908 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35181: skipped 000847 1787379356.736510 read 0x0202af thru 0x02be5e (0xbbb0 bytes) 000851 1787379356.763387 mapread 0x02c392 thru 0x034656 (0x82c5 bytes) 000853 1787379356.805865 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000854 1787379357.371678 mapread 0x02c00f thru 0x02d2cf (0x12c1 bytes) 000855 1787379357.449666 read 0x01c0e2 thru 0x01edbb (0x2cda bytes) 000861 1787379357.644730 write 0x01167b thru 0x01621a (0x4ba0 bytes) 000864 1787379357.779259 trunc from 0x035181 to 0x009205 (0x2bf7b bytes) 000865 1787379357.817421 trunc from 0x009206 to 0x01e53d (0x15338 bytes) 000870 1787379357.873245 write 0x03837a thru 0x03fb0f (0x7796 bytes) truncating to largest ever: 0x3f9dc 000871 1787379357.875958 trunc from 0x03fb10 to 0x03f9db (0x134 bytes) 000877 1787379357.920221 trunc from 0x03f9dc to 0x0006a1 (0x3f33a bytes) 000879 1787379357.979370 mapwrite 0x003acb thru 0x01268a (0xebc0 bytes) 000880 1787379358.207105 read 0x0076bf thru 0x01268a (0xafcc bytes) 000881 1787379358.207921 mapread 0x009aa9 thru 0x00f04e (0x55a6 bytes) 000882 1787379358.209103 read 0x00d52b thru 0x00eee7 (0x19bd bytes) 000886 1787379358.213959 mapwrite 0x00f80b thru 0x011f1a (0x2710 bytes) 000887 1787379358.248978 mapwrite 0x015632 thru 0x0188a3 (0x3272 bytes) 000888 1787379358.310632 MIRROR_RESYNC 2 mirrors 000889 1787379358.818451 mapwrite 0x0076da thru 0x00db52 (0x6479 bytes) 000891 1787379359.013648 write 0x03e728 thru 0x03ffff (0x18d8 bytes) 000892 1787379359.018184 read 0x029305 thru 0x035fb2 (0xccae bytes) 000893 1787379359.028116 write 0x01dab3 thru 0x026870 (0x8dbe bytes) 000894 1787379359.043061 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000895 1787379359.162340 trunc from 0x040000 to 0x014e78 (0x2b187 bytes) 000896 1787379359.223214 mapwrite 0x025b2d thru 0x031d78 (0xc24c bytes) 000904 1787379359.341730 MIRROR_EXTEND to 2 mirrors 000905 1787379359.849445 mapread 0x0309b5 thru 0x031d78 (0x13c4 bytes) 000908 1787379359.886277 MIRROR_RESYNC 2 mirrors 000911 1787379360.370388 mapread 0x03077d thru 0x031d78 (0x15fc bytes) 000912 1787379360.394054 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000913 1787379360.649956 MIRROR_EXTEND to 2 mirrors 000917 1787379361.445432 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000919 1787379361.604025 write 0x00a705 thru 0x00f124 (0x4a20 bytes) 000920 1787379361.633414 MIRROR_EXTEND to 2 mirrors 000922 1787379362.379765 write 0x01e31a thru 0x0279ec (0x96d3 bytes) 000923 1787379362.465561 mapread 0x014ad8 thru 0x0192ce (0x47f7 bytes) 000925 1787379362.482403 trunc from 0x031d79 to 0x00077d (0x315fb bytes) 000926 1787379362.504174 mapwrite 0x01b314 thru 0x01f5d1 (0x42be bytes) 000927 1787379362.582222 mapwrite 0x010c26 thru 0x012358 (0x1733 bytes) 000928 1787379362.627347 read 0x007613 thru 0x00a2f2 (0x2ce0 bytes) 000931 1787379362.638924 MIRROR_RESYNC 2 mirrors 000932 1787379363.141154 write 0x008ceb thru 0x018ca9 (0xffbf bytes) 000933 1787379363.207124 read 0x0052d6 thru 0x00d41b (0x8146 bytes) 000934 1787379363.213090 mapwrite 0x02174e thru 0x02d268 (0xbb1b bytes) 000935 1787379363.328209 write 0x009e20 thru 0x00da0e (0x3bef bytes) 000936 1787379363.329593 mapread 0x0001bb thru 0x0037c1 (0x3607 bytes) 000937 1787379363.348102 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2d269): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d269): Mirror 2: 0xb26f7e67 (stale) CRC-32 checksum value for chunk [0, 0x2d269): Mirror 1: 0x82dda7f5 Mirror 3: 0x82dda7f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d269: skipped 000938 1787379364.208419 mapread 0x005552 thru 0x011ac8 (0xc577 bytes) 000939 1787379364.260062 MIRROR_RESYNC 3 mirrors 000940 1787379364.928471 write 0x00aca7 thru 0x015ac6 (0xae20 bytes) 000941 1787379365.010636 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2d269): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d269): Mirror 2: 0x82dda7f5 (stale) Mirror 3: 0x82dda7f5 (stale) CRC-32 checksum value for chunk [0, 0x2d269): Mirror 1: 0x484869ca Mirror 4: 0x484869ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d269: skipped 000942 1787379365.773610 read 0x011bfd thru 0x016050 (0x4454 bytes) 000943 1787379365.788846 mapread 0x00720c thru 0x00e343 (0x7138 bytes) 000944 1787379365.831183 trunc from 0x02d269 to 0x0044af (0x28db9 bytes) 000945 1787379365.936839 read 0x00219a thru 0x0044af (0x2316 bytes) 000946 1787379365.947925 mapread 0x0009db thru 0x0044af (0x3ad5 bytes) 000948 1787379365.950818 read 0x00364d thru 0x0044af (0xe63 bytes) 000949 1787379365.956513 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x44b0): Mirror 3: 0xad340ac7 (stale) Mirror 4: 0xad340ac7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x44b0): Mirror 3: 0xad340ac7 (stale) Mirror 4: 0xad340ac7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x44b0: skipped 000952 1787379366.229899 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x44b0): Mirror 4: 0xad340ac7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x44b0): Mirror 4: 0xad340ac7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x44b0: skipped 000954 1787379366.500751 mapread 0x0027c0 thru 0x0044af (0x1cf0 bytes) 000956 1787379366.526357 mapwrite 0x007109 thru 0x00bcc8 (0x4bc0 bytes) 000959 1787379366.637372 mapread 0x00aff1 thru 0x00bcc8 (0xcd8 bytes) 000962 1787379366.642182 write 0x03abc4 thru 0x03ae24 (0x261 bytes) 000963 1787379366.649974 MIRROR_RESYNC 2 mirrors 000966 1787379367.227585 trunc from 0x03ae25 to 0x001785 (0x3969f bytes) 000967 1787379367.307530 read 0x00132c thru 0x001785 (0x45a bytes) 000968 1787379367.313470 trunc from 0x001786 to 0x007392 (0x5c0d bytes) 000970 1787379367.341991 write 0x02f897 thru 0x0329d4 (0x313e bytes) 000972 1787379367.344098 read 0x011b30 thru 0x015e2b (0x42fc bytes) 000975 1787379367.359234 mapwrite 0x01b9e0 thru 0x0257cc (0x9ded bytes) 000977 1787379367.418168 MIRROR_RESYNC 2 mirrors 000978 1787379368.024556 mapwrite 0x00c80c thru 0x01966f (0xce64 bytes) 000979 1787379368.190530 mapread 0x031708 thru 0x0329d4 (0x12cd bytes) 000982 1787379368.202421 trunc from 0x0329d5 to 0x0139dd (0x1eff7 bytes) 000983 1787379368.253113 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a0:0x0]: Invalid argument (22) 000984 1787379368.449485 trunc from 0x0139de to 0x005511 (0xe4cc bytes) 000987 1787379368.474089 mapwrite 0x031642 thru 0x03e76a (0xd129 bytes) 000989 1787379368.600331 trunc from 0x03e76b to 0x015b25 (0x28c45 bytes) 000990 1787379368.626509 mapread 0x008cfb thru 0x00d195 (0x449b bytes) 000991 1787379368.643014 mapread 0x009f48 thru 0x00f662 (0x571b bytes) 000992 1787379368.645241 trunc from 0x015b26 to 0x036641 (0x20b1c bytes) 000993 1787379368.678113 MIRROR_EXTEND to 2 mirrors 000994 1787379369.532721 read 0x0254c4 thru 0x025557 (0x94 bytes) 000996 1787379369.562038 MIRROR_RESYNC 2 mirrors 000997 1787379370.083716 write 0x036f60 thru 0x03ffff (0x90a0 bytes) 000998 1787379370.145114 write 0x02ce3f thru 0x03b5c1 (0xe783 bytes) 001000 1787379370.152202 mapread 0x003f08 thru 0x00973b (0x5834 bytes) All operations completed A-OK! PASS 70a (101s) == sanity-flr test 71: check mirror extend parent fid ==== 02:16:15 (1787379375) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 02:16:19 (1787379379) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.444033 s, 4.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147278 s, 7.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 02:16:30 (1787379390) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5109760 .. Extending file size to 1430624 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2680032 .. Extending file size to 7002272 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4093184 .. Extending file size to 6061312 .. Extending file size to 7694464 .. Extending file size to 8660160 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7832416 .. Extending file size to 8185376 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8797312 .. Extending file size to 8891136 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56916 56917 56919 56920 56921 failed 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:17:53 (1787379473) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8198976 .. Extending file size to 1741472 .. Extending file size to 6085920 .. Extending file size to 8813728 .. Extending file size to 9000704 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8969408 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9256384 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58409 58410 58412 58413 58417 done 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:19:12 (1787379552) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124384 s, 16.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.468516 s, 4.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xd61a8d68 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:19:34 (1787379574) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101519 s, 20.7 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 02:19:41 (1787379581) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f1:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x30c:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:19:49 (1787379589) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.235407 s, 8.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:19:57 (1787379597) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.369154 s, 5.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x30f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x310:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:20:06 (1787379606) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.351056 s, 6.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x311:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x312:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:20:38 (1787379638) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117756 s, 8.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f7:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:21:39 (1787379699) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259659 s, 8.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x316:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (65s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:22:43 (1787379763) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. Creating new pool test_204f oleg148-server: Pool lustre.test_204f created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.448838 s, 4.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x317:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2f9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg148-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:24:07 (1787379847) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:24:15 (1787379855) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:24:23 (1787379863) oleg148-server: Pool lustre.test_206 created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg148-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:24:48 (1787379888) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671144 s, 15.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447981 s, 23.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf8b822f Mirror 3: 0xf8b822f PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:25:01 (1787379901) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:25:06 (1787379906) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:25:11 (1787379911) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.141188 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x880000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2190000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2190000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2190000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1a00000 Read this (4 bytes): BBBB 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:25:54 (1787379954) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0361572 s, 227 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:26:02 (1787379962) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541322 s, 19.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x21d lmm_fid: [0x200000404:0x21d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x21d lmm_fid: [0x200000404:0x21d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834461 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 02:26:03.000000000 -0400 Modify: 2026-08-22 02:26:04.000000000 -0400 Change: 2026-08-22 02:26:05.000000000 -0400 Birth: 2026-08-22 02:26:03.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:26:11 (1787379971) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:26:14 (1787379974) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 1.11952 s, 36.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0825187 s, 49.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 22 02:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x21f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 22 02:26 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:26:25 (1787379985) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:26:34 (1787379994) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1778 sec =========== 02:26:45 (1787380005) === sanity-flr: start cleanup 02:26:47 (1787380007) === === sanity-flr: finish cleanup 02:26:53 (1787380013) ===