-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 19:32:12 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg208-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:32:31 (1787787151) === oleg208-client.virtnet: executing check_config_client /mnt/lustre oleg208-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg208-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c93d0b72800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c93d0b72800.idle_timeout=debug disable quota as required oleg208-server: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:33:09 (1787787189) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7627] == sanity-flr test 0a: lfs mirror create with -N option == 19:33:12 (1787787192) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:33:28 (1787787208) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:33:32 (1787787212) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:33:36 (1787787216) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:34:02 (1787787242) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:34:05 (1787787245) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 19:34:09 (1787787249) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c93d0b72800.stats=clear osc.lustre-OST0001-osc-ffff8c93d0b72800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:34:17 (1787787257) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:34:28 (1787787268) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:34:38 (1787787278) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:34:42 (1787787282) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:34:50 (1787787290) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 19:35:01 (1787787301) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:35:09 (1787787309) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:35:12 (1787787312) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:35:19 (1787787319) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:35:28 (1787787328) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:35:34 (1787787334) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.850095 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692462 s, 15.1 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:35:43 (1787787343) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.153699 s, 47.8 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:35:51 (1787787351) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:36:01 (1787787361) Stopping /mnt/lustre-ost1 (opts:) on oleg208-server oleg208-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c93d0b72800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c93d0b72800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg208-server: oleg208-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg208-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg208-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg208-server oleg208-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c93d0b72800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c93d0b72800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg208-server: oleg208-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg208-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (136s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:38:17 (1787787497) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.190584 s, 16.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg208-server oleg208-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c93d0b72800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c93d0b72800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg208-server: oleg208-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg208-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:39:01 (1787787541) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg208-server oleg208-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c93d0b72800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c93d0b72800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg208-server: oleg208-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg208-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg208-server oleg208-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c93d0b72800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c93d0b72800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg208-server: oleg208-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg208-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:40:16 (1787787616) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.67765 s, 11.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 19:40 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.180s user 0m0.000s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg208-server oleg208-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c93d0b72800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c93d0b72800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.234s user 0m0.001s sys 0m0.183s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg208-server: oleg208-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg208-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg208-server oleg208-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c93d0b72800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c93d0b72800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.116s user 0m0.000s sys 0m0.152s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg208-server: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg208-server: oleg208-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg208-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg208-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c93d0b72800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c93d0b72800.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:41:52 (1787787712) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:41:55 (1787787715) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:41:58 (1787787718) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 19:42:02 (1787787722) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 19:42:08 (1787787728) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.475445 s, 26.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:42:20 (1787787740) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656448 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654085 s, 16.0 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:42:27 (1787787747) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:42:34 (1787787754) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000732454 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116013 s, 883 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000750832 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113291 s, 904 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 19:42:44 (1787787764) create test files with size 15744974 .. deed5eda933f26027ae67d65357e6712 - 4660bb10437dbbd9a9b2bf0a83509682 - 91fbc3f992891a9bbf5c7d7c4c829a23 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg208-client.virtnet /mnt/lustre (opts:) Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 19:43:08 (1787787788) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:43:20 (1787787800) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:43:26 (1787787806) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135773 s, 7.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454563 s, 23.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 19:43:36 (1787787816) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133474 s, 15.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.29845 s, 14.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.218694 s, 9.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 19:43:50 (1787787830) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 19:43:52 (1787787832) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:43:55 (1787787835) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180024 s, 58.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 43c: read prefer uncompressed mirror == 19:44:02 (1787787842) llite.lustre-ffff8c93c2f26000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (6s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 19:44:08 (1787787848) llite.lustre-ffff8c93c2f26000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240226 s, 43.6 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145252 s, 72.2 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 19:44:16 (1787787856) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:44:19 (1787787859) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:44:29 (1787787869) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.320564 s, 32.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg208-client.virtnet /mnt/lustre (opts:) Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:44:41 (1787787881) llite.lustre-ffff8c93c4f59800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.224148 s, 46.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg208-client.virtnet /mnt/lustre (opts:) Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8c93d0c30000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:44:54 (1787787894) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:45:34 (1787787934) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (34s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:46:08 (1787787968) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 19:46:12 (1787787972) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.207611 s, 15.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.129042 s, 24.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xacde38bc (nosync,stale) Mirror 2: 0xacde38bc (nosync,stale) ** verify mirror 2 doesn't change original checksum: 17dfc01bb79b9ba7ac05f63cd425c1b4 - mirror 1 checksum: 365bffa406bd0df074d567a3e94e2e2f - mirror 2 checksum: 17dfc01bb79b9ba7ac05f63cd425c1b4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T19:46:14-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T19:46:14-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 17dfc01bb79b9ba7ac05f63cd425c1b4 - mirror 1 checksum: 365bffa406bd0df074d567a3e94e2e2f - mirror 2 checksum: 365bffa406bd0df074d567a3e94e2e2f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T19:46:14-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T19:46:14-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf0:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:46:26 (1787787986) Failing mds1 on oleg208-server Stopping /mnt/lustre-mds1 (opts:) on oleg208-server 19:46:32 (1787787992) shut down facet: mds1 facet_host: oleg208-server facet_failover_host: oleg208-server Failover mds1 to oleg208-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg208-server: oleg208-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg208-client: oleg208-server: ssh exited with exit code 1 Started lustre-MDT0000 19:46:54 (1787788014) targets are mounted 19:46:54 (1787788014) facet_failover done oleg208-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585409 s, 17.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf1:0x0] } PASS 50A (46s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:47:12 (1787788032) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:47:16 (1787788036) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:47:19 (1787788039) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00804484 s, 2.0 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:47:27 (1787788047) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:47:30 (1787788050) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0107759 s, 380 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0703201 s, 58.2 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:47:38 (1787788058) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:48:00 (1787788080) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:48:28 (1787788108) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:48:51 (1787788131) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:49:14 (1787788154) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 19:49:21 (1787788161) mirror create pid 55574 mirror split pid 55576 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55574 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55576 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 19:50:29 (1787788229) Seed set to 8230 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005 1787788230.990311 write 0x032059 thru 0x035955 (0x38fd bytes) truncating to largest ever: 0x252e6 000006 1787788230.993430 trunc from 0x035956 to 0x0252e5 (0x10670 bytes) 000007 1787788231.009984 read 0x01829f thru 0x01d6bf (0x5421 bytes) 000008 1787788231.020662 write 0x006f80 thru 0x0114fe (0xa57f bytes) 000009 1787788231.033008 write 0x007034 thru 0x008b84 (0x1b51 bytes) 000010 1787788231.034493 mapwrite 0x007801 thru 0x00ce80 (0x5680 bytes) 000013 1787788231.106091 read 0x00894f thru 0x017d32 (0xf3e4 bytes) truncating to largest ever: 0x35943 000014 1787788231.125800 trunc from 0x0252e6 to 0x035942 (0x1065d bytes) 000015 1787788231.143020 write 0x00ee73 thru 0x018930 (0x9abe bytes) 000016 1787788231.145070 read 0x02a3c3 thru 0x02d8fd (0x353b bytes) 000018 1787788231.159985 MIRROR_EXTEND to 2 mirrors 000019 1787788231.796262 mapwrite 0x02b978 thru 0x0348d3 (0x8f5c bytes) 000022 1787788231.994769 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000026 1787788232.158070 write 0x00576c thru 0x00b8de (0x6173 bytes) 000027 1787788232.173714 read 0x01d45c thru 0x023c66 (0x680b bytes) truncating to largest ever: 0x3a9ad 000028 1787788232.185013 trunc from 0x035943 to 0x03a9ac (0x506a bytes) 000030 1787788232.247077 trunc from 0x03a9ad to 0x003307 (0x376a5 bytes) 000031 1787788232.286730 trunc from 0x003308 to 0x0021bb (0x114c bytes) 000033 1787788232.317142 MIRROR_EXTEND to 2 mirrors 000038 1787788233.016097 mapread 0x001aa0 thru 0x0021bb (0x71c bytes) 000039 1787788233.025978 write 0x03fc1e thru 0x03ffff (0x3e2 bytes) 000040 1787788233.089439 mapread 0x017122 thru 0x026a19 (0xf8f8 bytes) 000042 1787788233.110562 MIRROR_RESYNC 2 mirrors 000043 1787788233.775693 mapwrite 0x030c14 thru 0x03769b (0x6a88 bytes) 000044 1787788233.939485 read 0x012eff thru 0x0222c5 (0xf3c7 bytes) 000047 1787788233.958771 read 0x01b8f4 thru 0x01ecd7 (0x33e4 bytes) 000048 1787788233.959722 read 0x028fb5 thru 0x02b68d (0x26d9 bytes) 000049 1787788233.976604 mapread 0x0292e2 thru 0x02ad8f (0x1aae bytes) 000051 1787788233.978043 mapread 0x03b9c1 thru 0x03ffff (0x463f bytes) 000052 1787788233.980980 read 0x0111f9 thru 0x018d3d (0x7b45 bytes) 000055 1787788233.983783 mapread 0x019a2c thru 0x0223b7 (0x898c bytes) 000056 1787788233.989644 trunc from 0x040000 to 0x019d8a (0x26275 bytes) 000057 1787788234.071520 mapread 0x0054a6 thru 0x012500 (0xd05b bytes) 000058 1787788234.072510 mapread 0x01433e thru 0x019d8a (0x5a4d bytes) 000059 1787788234.074050 trunc from 0x019d8b to 0x03152d (0x177a3 bytes) 000061 1787788234.099442 trunc from 0x03152e to 0x023a14 (0xdb19 bytes) 000063 1787788234.120697 trunc from 0x023a15 to 0x021f5b (0x1ab9 bytes) 000064 1787788234.144958 mapwrite 0x02b759 thru 0x031b61 (0x6409 bytes) 000065 1787788234.334608 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000070 1787788234.515573 mapread 0x019d44 thru 0x01bcc8 (0x1f85 bytes) 000072 1787788234.539555 trunc from 0x031b62 to 0x02bdf7 (0x5d6a bytes) 000074 1787788234.568595 write 0x032998 thru 0x03ffff (0xd668 bytes) 000075 1787788234.569673 mapwrite 0x037cbc thru 0x03ffff (0x8344 bytes) 000078 1787788234.637569 read 0x010cc6 thru 0x0208aa (0xfbe5 bytes) 000081 1787788234.650272 write 0x01c3f9 thru 0x01f743 (0x334b bytes) 000082 1787788234.651007 write 0x014795 thru 0x0246a2 (0xff0e bytes) 000083 1787788234.657260 mapwrite 0x019cd1 thru 0x0299bb (0xfceb bytes) 000084 1787788234.698763 trunc from 0x040000 to 0x011923 (0x2e6dc bytes) 000085 1787788234.707583 read 0x00705b thru 0x011923 (0xa8c9 bytes) 000086 1787788234.724726 MIRROR_EXTEND to 2 mirrors 000090 1787788235.321139 mapread 0x00d5f2 thru 0x011923 (0x4332 bytes) 000091 1787788235.340871 mapwrite 0x00fd71 thru 0x01572a (0x59ba bytes) 000092 1787788235.468560 mapwrite 0x00d995 thru 0x01c5af (0xec1b bytes) 000094 1787788235.548382 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c5b0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c5b0): Mirror 2: 0xd387549c (stale) CRC-32 checksum value for chunk [0, 0x1c5b0): Mirror 1: 0x64c5789f Mirror 3: 0x64c5789f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c5b0: skipped 000096 1787788236.149676 trunc from 0x01c5b0 to 0x00820b (0x143a4 bytes) 000097 1787788236.254820 mapwrite 0x02c7f1 thru 0x030b05 (0x4315 bytes) 000098 1787788236.330738 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30b06): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b06): Mirror 2: 0x1e21c768 (stale) Mirror 3: 0xe70f18fa (stale) CRC-32 checksum value for chunk [0, 0x30b06): Mirror 1: 0xacb91331 Mirror 4: 0xacb91331 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b06: skipped 000100 1787788237.014250 write 0x004e96 thru 0x0124bc (0xd627 bytes) 000101 1787788237.090680 trunc from 0x030b06 to 0x01234d (0x1e7b8 bytes) 000102 1787788237.137250 mapread 0x00bb09 thru 0x00d3d6 (0x18ce bytes) 000104 1787788237.139548 mapread 0x004c54 thru 0x00d320 (0x86cd bytes) 000105 1787788237.140607 mapwrite 0x01598a thru 0x024d28 (0xf39f bytes) 000107 1787788237.243477 mapread 0x000ef8 thru 0x00c337 (0xb440 bytes) 000109 1787788237.251065 mapwrite 0x022aaf thru 0x02c37d (0x98cf bytes) truncating to largest ever: 0x3be3d 000110 1787788237.310259 trunc from 0x02c37e to 0x03be3c (0xfabf bytes) 000111 1787788237.329653 trunc from 0x03be3d to 0x0302cf (0xbb6d bytes) 000112 1787788237.351792 trunc from 0x0302d0 to 0x037a2f (0x7760 bytes) 000113 1787788237.372418 mapwrite 0x01f74e thru 0x02b681 (0xbf34 bytes) 000114 1787788237.409285 mapwrite 0x0338c2 thru 0x03ffff (0xc73e bytes) 000115 1787788237.476321 write 0x026976 thru 0x030a04 (0xa08f bytes) 000116 1787788237.478124 write 0x01f17c thru 0x027418 (0x829d bytes) 000119 1787788237.481455 read 0x034314 thru 0x036b1f (0x280c bytes) 000120 1787788237.482369 trunc from 0x040000 to 0x006867 (0x39798 bytes) 000122 1787788237.507537 MIRROR_RESYNC 4 mirrors 000123 1787788238.271662 write 0x039f7f thru 0x03ffff (0x6081 bytes) 000125 1787788238.333741 write 0x00cac4 thru 0x0142d1 (0x780e bytes) 000128 1787788238.343231 mapread 0x029135 thru 0x02e711 (0x55dd bytes) 000129 1787788238.357977 write 0x01c1b7 thru 0x01f962 (0x37ac bytes) 000132 1787788238.367297 read 0x02bbbe thru 0x037339 (0xb77c bytes) 000133 1787788238.372569 write 0x005b0a thru 0x007aa3 (0x1f9a bytes) 000135 1787788238.379554 mapread 0x00d12d thru 0x01c27a (0xf14e bytes) 000136 1787788238.387029 mapwrite 0x013742 thru 0x01522d (0x1aec bytes) 000139 1787788238.441478 write 0x02a3f1 thru 0x03411d (0x9d2d bytes) 000140 1787788238.442682 trunc from 0x040000 to 0x013225 (0x2cdda bytes) 000141 1787788238.468964 trunc from 0x013226 to 0x0368c7 (0x236a2 bytes) 000143 1787788238.491557 MIRROR_RESYNC 4 mirrors 000144 1787788239.218613 mapread 0x00f97d thru 0x019127 (0x97ab bytes) 000148 1787788239.235793 mapwrite 0x03cee9 thru 0x03f126 (0x223e bytes) 000150 1787788239.360865 mapread 0x0232d5 thru 0x02deff (0xac2b bytes) 000151 1787788239.376814 mapread 0x00a910 thru 0x00ee71 (0x4562 bytes) 000153 1787788239.385903 read 0x023ae7 thru 0x0301e6 (0xc700 bytes) 000156 1787788239.387545 read 0x01e80b thru 0x02907f (0xa875 bytes) 000157 1787788239.388878 read 0x027047 thru 0x02ac31 (0x3beb bytes) 000158 1787788239.390138 mapwrite 0x03150a thru 0x03916a (0x7c61 bytes) 000159 1787788239.437822 trunc from 0x03f127 to 0x03b22e (0x3ef8 bytes) 000165 1787788239.463501 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b22f): Mirror 3: 0xa05735ea (stale) Mirror 4: 0xa05735ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b22f): Mirror 3: 0xa05735ea (stale) Mirror 4: 0xa05735ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b22f: skipped 000166 1787788239.721534 read 0x025a17 thru 0x027161 (0x174b bytes) 000167 1787788239.735608 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b22f): Mirror 4: 0xa05735ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b22f): Mirror 4: 0xa05735ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b22f: skipped 000168 1787788239.961213 read 0x009a62 thru 0x00d401 (0x39a0 bytes) 000169 1787788239.970937 mapread 0x023357 thru 0x028ccd (0x5977 bytes) 000173 1787788239.989378 mapread 0x0222b1 thru 0x02c52c (0xa27c bytes) 000174 1787788239.990556 mapread 0x035760 thru 0x035ffe (0x89f bytes) 000175 1787788240.001252 write 0x0016a6 thru 0x004c30 (0x358b bytes) 000180 1787788240.011450 trunc from 0x03b22f to 0x01578b (0x25aa3 bytes) 000181 1787788240.030895 write 0x02720f thru 0x036ae5 (0xf8d7 bytes) 000183 1787788240.034620 read 0x02ece8 thru 0x02ee53 (0x16c bytes) 000185 1787788240.041754 MIRROR_RESYNC 2 mirrors 000187 1787788240.584208 read 0x01e47a thru 0x023981 (0x5508 bytes) 000188 1787788240.604547 mapwrite 0x01e0e1 thru 0x025dba (0x7cda bytes) 000190 1787788240.793311 read 0x02f5d1 thru 0x033df1 (0x4821 bytes) 000191 1787788240.852317 mapwrite 0x037b21 thru 0x03ffff (0x84df bytes) 000193 1787788241.038268 write 0x02e231 thru 0x032e55 (0x4c25 bytes) 000195 1787788241.055021 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000196 1787788241.198073 trunc from 0x040000 to 0x026a71 (0x1958e bytes) 000197 1787788241.213485 read 0x01a2d7 thru 0x0204ab (0x61d5 bytes) 000199 1787788241.229369 MIRROR_EXTEND to 2 mirrors 000201 1787788241.835602 mapread 0x01a063 thru 0x023ffd (0x9f9b bytes) 000202 1787788241.850468 mapwrite 0x020bdd thru 0x02e4d5 (0xd8f9 bytes) 000203 1787788242.097885 trunc from 0x02e4d6 to 0x012742 (0x1bd93 bytes) 000204 1787788242.125339 mapread 0x00ab98 thru 0x012742 (0x7bab bytes) 000205 1787788242.135952 read 0x005192 thru 0x00d613 (0x8482 bytes) 000206 1787788242.140487 MIRROR_RESYNC 2 mirrors 000209 1787788242.651956 read 0x003b27 thru 0x010c66 (0xd140 bytes) 000210 1787788242.661230 read 0x011798 thru 0x012742 (0xfab bytes) 000211 1787788242.670738 trunc from 0x012743 to 0x01ba74 (0x9332 bytes) 000212 1787788242.767257 mapread 0x00fcbe thru 0x01292f (0x2c72 bytes) 000214 1787788242.781429 mapwrite 0x01b560 thru 0x02769c (0xc13d bytes) 000215 1787788242.881467 mapwrite 0x01283a thru 0x01d224 (0xa9eb bytes) 000216 1787788242.946763 trunc from 0x02769d to 0x000deb (0x268b1 bytes) 000219 1787788242.958740 trunc from 0x000dec to 0x035d50 (0x34f65 bytes) 000220 1787788242.982105 mapread 0x02c4b5 thru 0x032bfe (0x674a bytes) 000222 1787788242.996542 mapwrite 0x0352a7 thru 0x03ffff (0xad59 bytes) 000223 1787788243.082459 write 0x001e52 thru 0x00266a (0x819 bytes) 000224 1787788243.092838 mapwrite 0x0007fb thru 0x0097d6 (0x8fdc bytes) 000225 1787788243.162750 write 0x01b98e thru 0x029e0d (0xe480 bytes) 000226 1787788243.169003 MIRROR_RESYNC 2 mirrors 000227 1787788243.755977 MIRROR_RESYNC 2 mirrors 000231 1787788244.052604 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000234 1787788244.157988 mapwrite 0x00d709 thru 0x0108c2 (0x31ba bytes) truncating to largest ever: 0x3e6ed 000236 1787788244.246856 trunc from 0x040000 to 0x03e6ec (0x1913 bytes) 000237 1787788244.306017 trunc from 0x03e6ed to 0x023eef (0x1a7fd bytes) 000238 1787788244.329167 mapwrite 0x01f0c6 thru 0x026eb1 (0x7dec bytes) 000240 1787788244.416093 MIRROR_EXTEND to 2 mirrors 000244 1787788244.921236 MIRROR_RESYNC 2 mirrors 000246 1787788245.233932 trunc from 0x026eb2 to 0x02306b (0x3e46 bytes) 000247 1787788245.358021 read 0x00b0c2 thru 0x01a9e0 (0xf91f bytes) 000248 1787788245.408592 MIRROR_RESYNC 2 mirrors 000250 1787788246.615067 mapwrite 0x00e409 thru 0x01cc41 (0xe839 bytes) 000252 1787788246.865207 mapread 0x0087ec thru 0x00aa83 (0x2298 bytes) 000253 1787788246.866546 write 0x012f8e thru 0x0192a4 (0x6317 bytes) 000254 1787788246.868492 mapwrite 0x01b3d3 thru 0x02543d (0xa06b bytes) 000257 1787788246.945645 mapwrite 0x0313d9 thru 0x0346ee (0x3316 bytes) 000259 1787788247.020383 mapread 0x0194e4 thru 0x021181 (0x7c9e bytes) 000260 1787788247.022615 read 0x007315 thru 0x00d7f1 (0x64dd bytes) 000261 1787788247.028097 write 0x01b717 thru 0x01be1c (0x706 bytes) 000262 1787788247.029005 mapwrite 0x02a678 thru 0x039adb (0xf464 bytes) 000263 1787788247.102517 read 0x01e32a thru 0x01ffd1 (0x1ca8 bytes) 000264 1787788247.103350 write 0x01a204 thru 0x01ba6f (0x186c bytes) 000267 1787788247.107548 trunc from 0x039adc to 0x0039d1 (0x3610a bytes) 000268 1787788247.127232 mapread 0x000473 thru 0x000bf0 (0x77e bytes) 000271 1787788247.135492 mapwrite 0x030c7e thru 0x039bc5 (0x8f48 bytes) 000272 1787788247.223798 write 0x0000b0 thru 0x009637 (0x9588 bytes) 000273 1787788247.226671 mapwrite 0x02530d thru 0x0297ec (0x44e0 bytes) 000274 1787788247.283095 MIRROR_RESYNC 2 mirrors 000277 1787788247.893277 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000278 1787788248.041201 trunc from 0x039bc6 to 0x005355 (0x34870 bytes) 000279 1787788248.092332 write 0x0102fb thru 0x01e451 (0xe157 bytes) 000281 1787788248.094671 mapread 0x00e645 thru 0x015805 (0x71c1 bytes) 000283 1787788248.105917 trunc from 0x01e452 to 0x036587 (0x18136 bytes) 000285 1787788248.127434 read 0x0257b4 thru 0x02d767 (0x7fb4 bytes) 000286 1787788248.139533 read 0x004333 thru 0x012a06 (0xe6d4 bytes) 000287 1787788248.149697 read 0x0019d4 thru 0x009b42 (0x816f bytes) 000291 1787788248.152136 mapread 0x018725 thru 0x018b39 (0x415 bytes) 000292 1787788248.159521 MIRROR_EXTEND to 2 mirrors 000293 1787788248.893579 MIRROR_RESYNC 2 mirrors 000294 1787788249.249248 write 0x03c818 thru 0x03ffff (0x37e8 bytes) 000295 1787788249.331183 trunc from 0x040000 to 0x036769 (0x9896 bytes) 000296 1787788249.392920 read 0x026893 thru 0x033e23 (0xd591 bytes) 000298 1787788249.405984 read 0x03486a thru 0x036769 (0x1f00 bytes) 000299 1787788249.418797 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3676a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3676a): Mirror 2: 0x3d0ae7e7 (stale) CRC-32 checksum value for chunk [0, 0x3676a): Mirror 1: 0x3d0ae7e7 Mirror 3: 0x3d0ae7e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3676a: skipped 000300 1787788250.283160 read 0x02672e thru 0x032bd1 (0xc4a4 bytes) 000303 1787788250.302932 mapread 0x00978f thru 0x0189ed (0xf25f bytes) 000305 1787788250.334860 trunc from 0x03676a to 0x039b81 (0x3418 bytes) 000306 1787788250.468339 mapwrite 0x0094db thru 0x013077 (0x9b9d bytes) 000307 1787788250.581495 trunc from 0x039b82 to 0x00fe8b (0x29cf6 bytes) 000309 1787788250.600093 write 0x03ab23 thru 0x03f298 (0x4776 bytes) 000310 1787788250.601336 read 0x03d2a1 thru 0x03f298 (0x1ff8 bytes) 000311 1787788250.603550 trunc from 0x03f299 to 0x01f70d (0x1fb8b bytes) 000312 1787788250.632585 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1f70e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f70e): Mirror 2: 0x7b19609a (stale) Mirror 3: 0x7b19609a (stale) CRC-32 checksum value for chunk [0, 0x1f70e): Mirror 1: 0x98fb716a Mirror 4: 0x98fb716a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f70e: skipped 000313 1787788251.451462 read 0x0115d0 thru 0x012540 (0xf71 bytes) 000315 1787788251.465863 trunc from 0x01f70e to 0x0083af (0x1735e bytes) 000317 1787788251.552630 mapwrite 0x0311ff thru 0x0354f2 (0x42f4 bytes) 000318 1787788251.617727 read 0x017712 thru 0x0196e3 (0x1fd2 bytes) 000319 1787788251.638544 mapread 0x004dd4 thru 0x008aa4 (0x3cd1 bytes) 000320 1787788251.646003 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x354f3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x354f3): Mirror 2: 0xa3e37ef6 (stale) Mirror 3: 0xa3e37ef6 (stale) Mirror 4: 0x7673c8e8 (stale) CRC-32 checksum value for chunk [0, 0x354f3): Mirror 1: 0xa187efe4 Mirror 5: 0xa187efe4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x354f3: skipped 000322 1787788252.529561 read 0x01acb9 thru 0x022831 (0x7b79 bytes) 000323 1787788252.545246 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x354f3): Mirror 3: 0xa3e37ef6 (stale) Mirror 4: 0x7673c8e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x354f3): Mirror 3: 0xa3e37ef6 (stale) Mirror 4: 0x7673c8e8 (stale) CRC-32 checksum value for chunk [0, 0x354f3): Mirror 1: 0xa187efe4 Mirror 5: 0xa187efe4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x354f3: skipped 000328 1787788253.022281 read 0x01fc2a thru 0x02e053 (0xe42a bytes) 000329 1787788253.030832 trunc from 0x0354f3 to 0x03d9cd (0x84db bytes) 000330 1787788253.167574 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d9ce): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9ce): Mirror 3: 0x8ea9be59 (stale) Mirror 4: 0x8204a694 (stale) Mirror 5: 0x5d17d29e (stale) CRC-32 checksum value for chunk [0, 0x3d9ce): Mirror 1: 0x5d17d29e Mirror 6: 0x5d17d29e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9ce: skipped 000331 1787788254.253351 read 0x01b1d0 thru 0x01f602 (0x4433 bytes) 000332 1787788254.261827 trunc from 0x03d9ce to 0x032d3e (0xac8f bytes) 000333 1787788254.378696 write 0x0036c7 thru 0x00b78f (0x80c9 bytes) 000336 1787788254.406517 trunc from 0x032d3f to 0x039898 (0x6b5a bytes) 000339 1787788254.433462 mapread 0x02ce99 thru 0x030f23 (0x408b bytes) 000343 1787788254.453453 mapread 0x00cfd2 thru 0x012924 (0x5953 bytes) 000345 1787788254.466580 write 0x00124e thru 0x00cf7f (0xbd32 bytes) 000347 1787788254.475072 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39899): Mirror 4: 0xd0153c51 (stale) Mirror 5: 0x44c75baa (stale) Mirror 6: 0x44c75baa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39899): Mirror 4: 0xd0153c51 (stale) Mirror 5: 0x44c75baa (stale) Mirror 6: 0x44c75baa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39899: skipped 000348 1787788254.732488 mapread 0x003a8e thru 0x0115cd (0xdb40 bytes) 000349 1787788254.781698 write 0x032194 thru 0x03e39e (0xc20b bytes) 000350 1787788254.798689 mapread 0x013ecf thru 0x01c940 (0x8a72 bytes) 000351 1787788254.814551 trunc from 0x03e39f to 0x0399bb (0x49e3 bytes) 000353 1787788254.871251 MIRROR_RESYNC 4 mirrors 000358 1787788255.947723 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000360 1787788256.210574 read 0x01f6e1 thru 0x028ae6 (0x9406 bytes) 000363 1787788256.228785 MIRROR_RESYNC 3 mirrors 000365 1787788256.511655 mapread 0x002c1f thru 0x010978 (0xdd5a bytes) 000366 1787788256.529292 mapread 0x029788 thru 0x030d0b (0x7584 bytes) 000367 1787788256.538861 write 0x0266cf thru 0x02f6b5 (0x8fe7 bytes) 000369 1787788256.608687 read 0x0292f4 thru 0x02f9ae (0x66bb bytes) 000370 1787788256.609749 mapread 0x003079 thru 0x00ea4a (0xb9d2 bytes) 000374 1787788256.621151 mapread 0x0237c6 thru 0x03113e (0xd979 bytes) 000375 1787788256.637522 mapwrite 0x03d2cd thru 0x03ffff (0x2d33 bytes) 000379 1787788256.730194 trunc from 0x040000 to 0x03236d (0xdc92 bytes) 000380 1787788256.757615 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3236e): Mirror 6: 0x4d59da5b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3236e): Mirror 6: 0x4d59da5b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3236e: skipped 000383 1787788257.000003 write 0x013e07 thru 0x022919 (0xeb13 bytes) 000385 1787788257.009633 trunc from 0x03236e to 0x038a5d (0x66f0 bytes) 000386 1787788257.033120 mapwrite 0x00982f thru 0x012bbd (0x938f bytes) 000387 1787788257.109712 trunc from 0x038a5e to 0x0265e9 (0x12474 bytes) 000389 1787788257.128282 mapwrite 0x036bd8 thru 0x03ffff (0x9428 bytes) 000390 1787788257.204852 trunc from 0x040000 to 0x01b024 (0x24fdb bytes) 000392 1787788257.223187 trunc from 0x01b025 to 0x02324a (0x8226 bytes) 000394 1787788257.237893 mapread 0x00d480 thru 0x01d0a7 (0xfc28 bytes) 000396 1787788257.248201 read 0x0034a4 thru 0x009947 (0x64a4 bytes) 000397 1787788257.252894 write 0x03a813 thru 0x03e9a3 (0x4191 bytes) 000398 1787788257.254318 mapread 0x02497c thru 0x02695b (0x1fe0 bytes) 000399 1787788257.265725 write 0x03a4cc thru 0x03ffff (0x5b34 bytes) 000400 1787788257.266936 mapwrite 0x0054bd thru 0x00f760 (0xa2a4 bytes) 000402 1787788257.314830 read 0x014985 thru 0x01f259 (0xa8d5 bytes) 000403 1787788257.315586 write 0x025c6e thru 0x02b461 (0x57f4 bytes) 000405 1787788257.318014 read 0x00f83a thru 0x010f51 (0x1718 bytes) 000407 1787788257.319060 write 0x02d504 thru 0x03a866 (0xd363 bytes) 000410 1787788257.336711 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000411 1787788257.530064 read 0x021853 thru 0x02c539 (0xace7 bytes) 000412 1787788257.543007 trunc from 0x040000 to 0x01d6d6 (0x22929 bytes) 000414 1787788257.558101 write 0x02461f thru 0x02cd66 (0x8748 bytes) 000415 1787788257.560646 mapwrite 0x02377f thru 0x02adde (0x7660 bytes) 000416 1787788257.602242 mapread 0x0040e9 thru 0x008f47 (0x4e5f bytes) 000417 1787788257.613667 trunc from 0x02cd67 to 0x01e5d3 (0xe793 bytes) 000419 1787788257.630141 mapwrite 0x012551 thru 0x01da28 (0xb4d8 bytes) 000420 1787788257.682374 mapwrite 0x0094d0 thru 0x01411f (0xac50 bytes) 000422 1787788257.744965 mapwrite 0x018fc5 thru 0x01fab7 (0x6af3 bytes) 000423 1787788257.796613 mapwrite 0x008e0a thru 0x0178ca (0xeac1 bytes) 000424 1787788257.847347 MIRROR_EXTEND to 2 mirrors 000425 1787788258.362233 trunc from 0x01fab8 to 0x0320ee (0x12637 bytes) 000426 1787788258.454926 read 0x022faa thru 0x0292a1 (0x62f8 bytes) 000427 1787788258.476733 write 0x013c15 thru 0x016e77 (0x3263 bytes) 000429 1787788258.490284 read 0x0289df thru 0x0320ee (0x9710 bytes) 000431 1787788258.503556 write 0x01c7c8 thru 0x027cff (0xb538 bytes) 000433 1787788258.509837 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x320ef): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x320ef): Mirror 2: 0x2a1cd5c0 (stale) CRC-32 checksum value for chunk [0, 0x320ef): Mirror 1: 0x15a85b82 Mirror 3: 0x15a85b82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x320ef: skipped 000436 1787788259.676240 mapread 0x022cce thru 0x026746 (0x3a79 bytes) 000439 1787788259.700232 trunc from 0x0320ef to 0x00552c (0x2cbc2 bytes) 000440 1787788259.806327 write 0x0010f3 thru 0x00c4b9 (0xb3c7 bytes) 000441 1787788259.811775 write 0x009526 thru 0x0147a7 (0xb282 bytes) 000442 1787788259.813140 mapwrite 0x02827e thru 0x02e0e7 (0x5e6a bytes) 000443 1787788259.896029 trunc from 0x02e0e8 to 0x01cc09 (0x114de bytes) 000444 1787788259.913498 mapread 0x009f12 thru 0x014297 (0xa386 bytes) 000445 1787788259.916644 write 0x00ae75 thru 0x01728a (0xc416 bytes) 000446 1787788259.924777 mapwrite 0x00424e thru 0x00f19f (0xaf52 bytes) 000447 1787788259.996710 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cc0a): Mirror 3: 0x7dc2e134 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cc0a): Mirror 3: 0x7dc2e134 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cc0a: skipped 000450 1787788260.397241 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1cc0a): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cc0a): Mirror 3: 0x7dc2e134 (stale) CRC-32 checksum value for chunk [0, 0x1cc0a): Mirror 1: 0x8103010d Mirror 4: 0x8103010d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cc0a: skipped 000452 1787788261.418231 write 0x0392a7 thru 0x03cfc6 (0x3d20 bytes) 000457 1787788261.510536 trunc from 0x03cfc7 to 0x01b194 (0x21e32 bytes) 000460 1787788261.555654 mapwrite 0x01a2d4 thru 0x01bc27 (0x1954 bytes) 000461 1787788261.649657 mapread 0x017257 thru 0x01bc27 (0x49d1 bytes) 000463 1787788261.661478 mapread 0x00c308 thru 0x012b0b (0x6804 bytes) 000464 1787788261.673509 write 0x01ecbe thru 0x028862 (0x9ba5 bytes) 000467 1787788261.677948 mapwrite 0x02b83a thru 0x03582b (0x9ff2 bytes) 000468 1787788261.869802 read 0x022c63 thru 0x02786d (0x4c0b bytes) 000471 1787788261.873306 read 0x0052d0 thru 0x006015 (0xd46 bytes) 000472 1787788261.874212 write 0x01ea23 thru 0x021bd0 (0x31ae bytes) 000474 1787788261.878274 write 0x02c398 thru 0x02ea54 (0x26bd bytes) 000475 1787788261.880429 write 0x0138b5 thru 0x01a71f (0x6e6b bytes) 000476 1787788261.882453 mapread 0x021673 thru 0x02e0e7 (0xca75 bytes) 000477 1787788261.884258 write 0x016c49 thru 0x023724 (0xcadc bytes) 000479 1787788261.887530 mapread 0x0057fb thru 0x013bc4 (0xe3ca bytes) 000482 1787788261.890454 mapwrite 0x0159f0 thru 0x01f061 (0x9672 bytes) 000483 1787788261.955684 write 0x008d4a thru 0x00cebe (0x4175 bytes) 000484 1787788261.978467 MIRROR_RESYNC 3 mirrors 000485 1787788262.738106 MIRROR_EXTEND to 4 mirrors 000488 1787788263.295324 read 0x030d2c thru 0x03582b (0x4b00 bytes) 000489 1787788263.302809 mapread 0x0150f4 thru 0x0244ec (0xf3f9 bytes) 000491 1787788263.329599 trunc from 0x03582c to 0x03d28b (0x7a60 bytes) 000493 1787788263.417613 trunc from 0x03d28c to 0x008440 (0x34e4b bytes) 000494 1787788263.441132 mapwrite 0x027061 thru 0x02a7cd (0x376d bytes) 000496 1787788263.505901 mapread 0x022bee thru 0x02867a (0x5a8d bytes) 000499 1787788263.512566 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a7ce): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a7ce): Mirror 3: 0x7cf97752 (stale) Mirror 4: 0x7cf97752 (stale) Mirror 5: 0x7cf97752 (stale) CRC-32 checksum value for chunk [0, 0x2a7ce): Mirror 1: 0x51473c62 Mirror 6: 0x51473c62 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a7ce: skipped 000501 1787788264.240928 mapwrite 0x036446 thru 0x03ffff (0x9bba bytes) 000502 1787788264.400441 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x47014495 (stale) Mirror 5: 0x47014495 (stale) Mirror 6: 0x60f056fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x47014495 (stale) Mirror 5: 0x47014495 (stale) Mirror 6: 0x60f056fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000505 1787788264.681946 trunc from 0x040000 to 0x03631b (0x9ce4 bytes) 000507 1787788264.702052 read 0x0099f5 thru 0x019316 (0xf922 bytes) 000508 1787788264.710268 mapread 0x010b2a thru 0x018595 (0x7a6c bytes) 000509 1787788264.711380 read 0x01b2d0 thru 0x023654 (0x8385 bytes) 000512 1787788264.723020 mapwrite 0x00b685 thru 0x010f49 (0x58c5 bytes) 000515 1787788264.767262 mapread 0x02dcb7 thru 0x02e28c (0x5d6 bytes) 000518 1787788264.770525 trunc from 0x03631c to 0x01f703 (0x16c18 bytes) 000520 1787788264.789780 trunc from 0x01f704 to 0x002e80 (0x1c883 bytes) 000523 1787788264.808486 write 0x013ada thru 0x0231e0 (0xf707 bytes) 000524 1787788264.811793 mapread 0x015ef6 thru 0x01adcd (0x4ed8 bytes) 000525 1787788264.813178 mapread 0x00debf thru 0x0176a8 (0x97ea bytes) 000528 1787788264.836730 trunc from 0x0231e1 to 0x00fe51 (0x1338f bytes) 000530 1787788264.853293 read 0x0052b7 thru 0x007e25 (0x2b6f bytes) 000531 1787788264.861167 mapwrite 0x00c6df thru 0x017aae (0xb3d0 bytes) 000532 1787788264.928190 write 0x02ec01 thru 0x032f57 (0x4357 bytes) 000534 1787788264.929830 trunc from 0x032f58 to 0x0184d7 (0x1aa80 bytes) 000538 1787788264.947605 mapwrite 0x01e22b thru 0x02a8e6 (0xc6bc bytes) 000539 1787788265.016814 trunc from 0x02a8e7 to 0x0130a1 (0x17845 bytes) 000541 1787788265.037594 read 0x00ccb7 thru 0x010562 (0x38ac bytes) 000542 1787788265.039287 read 0x009001 thru 0x00b22f (0x222f bytes) 000543 1787788265.040758 trunc from 0x0130a2 to 0x033e22 (0x20d81 bytes) 000545 1787788265.059086 trunc from 0x033e23 to 0x017b9c (0x1c286 bytes) 000548 1787788265.075168 trunc from 0x017b9d to 0x002c56 (0x14f46 bytes) 000549 1787788265.099455 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c57): Mirror 4: 0x9ba9f9f6 (stale) Mirror 6: 0x9ba9f9f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c57): Mirror 4: 0x9ba9f9f6 (stale) Mirror 6: 0x9ba9f9f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c57: skipped 000550 1787788265.325768 trunc from 0x002c57 to 0x00a054 (0x73fe bytes) 000551 1787788265.346076 write 0x00c4dd thru 0x014ca0 (0x87c4 bytes) 000552 1787788265.347544 mapwrite 0x02c64b thru 0x033e58 (0x780e bytes) 000558 1787788265.421276 write 0x02aa2a thru 0x0338ca (0x8ea1 bytes) 000561 1787788265.424535 write 0x011161 thru 0x01f340 (0xe1e0 bytes) 000562 1787788265.430043 mapread 0x02292c thru 0x029865 (0x6f3a bytes) 000566 1787788265.440261 read 0x0106d7 thru 0x0125a8 (0x1ed2 bytes) 000570 1787788265.442449 mapread 0x00ef26 thru 0x0165bd (0x7698 bytes) 000572 1787788265.444488 mapwrite 0x0185f0 thru 0x01a2e6 (0x1cf7 bytes) 000574 1787788265.489885 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33e59): Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e59): Mirror 4: 0x15e96f0c (stale) Mirror 6: 0x48e1584d (stale) CRC-32 checksum value for chunk [0, 0x33e59): Mirror 1: 0x6876bc5a Mirror 7: 0x6876bc5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e59: skipped 000575 1787788266.173637 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33e59): Mirror 6: 0x48e1584d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e59): Mirror 6: 0x48e1584d (stale) CRC-32 checksum value for chunk [0, 0x33e59): Mirror 1: 0x6876bc5a Mirror 7: 0x6876bc5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e59: skipped 000577 1787788266.563896 mapread 0x019d5c thru 0x0271cd (0xd472 bytes) 000578 1787788266.591725 trunc from 0x033e59 to 0x01ca52 (0x17406 bytes) 000579 1787788266.701303 mapread 0x019f7c thru 0x01af59 (0xfde bytes) 000581 1787788266.715657 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ca53): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ca53): Mirror 6: 0x9273e871 (stale) Mirror 7: 0x14b86b48 (stale) CRC-32 checksum value for chunk [0, 0x1ca53): Mirror 1: 0x14b86b48 Mirror 8: 0x14b86b48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ca53: skipped 000585 1787788267.389287 write 0x01d016 thru 0x02a793 (0xd77e bytes) 000586 1787788267.460516 mapwrite 0x027dd7 thru 0x02f525 (0x774f bytes) 000587 1787788267.537877 trunc from 0x02f526 to 0x029e46 (0x56df bytes) 000588 1787788267.558891 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29e47): Mirror 7: 0xe1d03ca6 (stale) Mirror 8: 0xbe2ea3b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29e47): Mirror 7: 0xe1d03ca6 (stale) Mirror 8: 0xbe2ea3b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29e47: skipped 000589 1787788267.821643 read 0x0020d9 thru 0x00cbbd (0xaae5 bytes) 000590 1787788267.830725 trunc from 0x029e47 to 0x010c0c (0x1923a bytes) 000593 1787788267.850061 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10c0d): Mirror 8: 0x9546e635 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10c0d): Mirror 8: 0x9546e635 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10c0d: skipped 000595 1787788268.062077 mapwrite 0x01b0ef thru 0x020b7d (0x5a8f bytes) 000596 1787788268.131276 trunc from 0x020b7e to 0x0263dc (0x585f bytes) 000597 1787788268.148679 read 0x00b543 thru 0x01a535 (0xeff3 bytes) 000599 1787788268.158680 trunc from 0x0263dd to 0x026fae (0xbd2 bytes) 000600 1787788268.180708 read 0x01d94e thru 0x022d2a (0x53dd bytes) 000603 1787788268.195463 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26faf): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26faf): Mirror 8: 0x10b15a76 (stale) CRC-32 checksum value for chunk [0, 0x26faf): Mirror 1: 0x111aebfb Mirror 9: 0x111aebfb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26faf: skipped 000605 1787788268.989517 mapwrite 0x02ac67 thru 0x02d6c1 (0x2a5b bytes) 000606 1787788269.235270 trunc from 0x02d6c2 to 0x0323c9 (0x4d08 bytes) 000608 1787788269.276466 trunc from 0x0323ca to 0x023c1c (0xe7ad bytes) 000609 1787788269.297614 MIRROR_RESYNC 3 mirrors 000610 1787788270.058285 mapread 0x009a3b thru 0x01769c (0xdc62 bytes) 000611 1787788270.084752 mapwrite 0x0229ea thru 0x02b2d1 (0x88e8 bytes) 000613 1787788270.229519 trunc from 0x02b2d2 to 0x00f627 (0x1bcaa bytes) 000615 1787788270.252289 mapread 0x00b9b4 thru 0x00f627 (0x3c74 bytes) 000616 1787788270.259978 trunc from 0x00f628 to 0x036cfa (0x276d3 bytes) 000618 1787788270.281578 mapwrite 0x0249d0 thru 0x02b89a (0x6ecb bytes) 000619 1787788270.335126 write 0x006204 thru 0x01180c (0xb609 bytes) 000620 1787788270.340683 trunc from 0x036cfb to 0x017ebd (0x1ee3d bytes) 000622 1787788270.362131 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17ebe): Mirror 9: 0x4c656541 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17ebe): Mirror 9: 0x4c656541 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17ebe: skipped 000624 1787788270.615276 write 0x02be49 thru 0x035c48 (0x9e00 bytes) 000625 1787788270.618641 trunc from 0x035c49 to 0x01bebe (0x19d8a bytes) 000628 1787788270.635533 mapread 0x012296 thru 0x01aedf (0x8c4a bytes) 000629 1787788270.653748 read 0x017b32 thru 0x01bebe (0x438d bytes) 000630 1787788270.654300 mapwrite 0x01b904 thru 0x0221d5 (0x68d2 bytes) 000631 1787788270.733852 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) 000634 1787788270.856703 write 0x02bce7 thru 0x02fcee (0x4008 bytes) 000636 1787788270.859230 mapwrite 0x020421 thru 0x022552 (0x2132 bytes) 000638 1787788270.925740 mapread 0x005d47 thru 0x00a7ab (0x4a65 bytes) 000640 1787788270.942714 mapread 0x01ce96 thru 0x024324 (0x748f bytes) 000641 1787788270.944197 mapwrite 0x01cc4f thru 0x028e83 (0xc235 bytes) 000642 1787788270.996675 mapwrite 0x03e3b4 thru 0x03ffff (0x1c4c bytes) 000644 1787788271.063176 mapread 0x027eb6 thru 0x029bf4 (0x1d3f bytes) 000646 1787788271.064861 trunc from 0x040000 to 0x0359f0 (0xa60f bytes) 000649 1787788271.086172 mapwrite 0x01fcc8 thru 0x0293b4 (0x96ed bytes) 000650 1787788271.131854 write 0x01b354 thru 0x01bf51 (0xbfe bytes) 000651 1787788271.134449 mapwrite 0x0193cd thru 0x025438 (0xc06c bytes) 000652 1787788271.204252 trunc from 0x0359f1 to 0x021868 (0x14188 bytes) 000653 1787788271.222806 write 0x035033 thru 0x03dcc2 (0x8c90 bytes) 000654 1787788271.225757 mapread 0x015a09 thru 0x01b178 (0x5770 bytes) 000657 1787788271.230505 mapwrite 0x03e28c thru 0x03ffff (0x1d74 bytes) 000658 1787788271.297524 write 0x025932 thru 0x0291ea (0x38b9 bytes) 000664 1787788271.309660 trunc from 0x040000 to 0x02b7f9 (0x14806 bytes) 000666 1787788271.334062 trunc from 0x02b7fa to 0x006c3f (0x24bba bytes) 000668 1787788271.352764 trunc from 0x006c40 to 0x00d5ae (0x696f bytes) 000670 1787788271.376003 MIRROR_EXTEND to 2 mirrors 000671 1787788271.876600 trunc from 0x00d5af to 0x020477 (0x12ec9 bytes) 000673 1787788271.953231 read 0x018930 thru 0x020477 (0x7b48 bytes) 000674 1787788271.960530 write 0x002ec4 thru 0x00f9ea (0xcb27 bytes) 000675 1787788271.967976 mapwrite 0x03748a thru 0x03ffff (0x8b76 bytes) 000676 1787788272.049005 write 0x005389 thru 0x00b441 (0x60b9 bytes) 000677 1787788272.050762 mapread 0x02ae83 thru 0x0300c6 (0x5244 bytes) 000678 1787788272.068294 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3fc6a404 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe4f91a95 Mirror 3: 0xe4f91a95 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000680 1787788272.799870 write 0x02150a thru 0x02d074 (0xbb6b bytes) 000683 1787788272.884616 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3fc6a404 (stale) Mirror 3: 0xe4f91a95 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x16154b4c Mirror 4: 0x16154b4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000686 1787788273.780108 MIRROR_RESYNC 4 mirrors 000687 1787788274.668316 MIRROR_EXTEND to 5 mirrors 000688 1787788275.511358 trunc from 0x040000 to 0x03c242 (0x3dbd bytes) 000691 1787788275.609630 mapwrite 0x03ef6e thru 0x03ffff (0x1092 bytes) 000693 1787788275.747247 mapwrite 0x00b1aa thru 0x012e43 (0x7c9a bytes) 000694 1787788275.813436 trunc from 0x040000 to 0x01aa53 (0x255ac bytes) 000695 1787788275.833374 write 0x03e565 thru 0x03ffff (0x1a9b bytes) 000696 1787788275.835368 mapread 0x004327 thru 0x00ce6d (0x8b47 bytes) 000700 1787788275.840255 mapread 0x015704 thru 0x021825 (0xc122 bytes) 000704 1787788275.878901 mapwrite 0x021b2d thru 0x03149e (0xf972 bytes) 000705 1787788276.017311 MIRROR_RESYNC 5 mirrors 000706 1787788277.106100 trunc from 0x040000 to 0x027965 (0x1869a bytes) 000707 1787788277.194024 read 0x01a256 thru 0x024fc0 (0xad6b bytes) 000708 1787788277.206308 read 0x013bb2 thru 0x01d99f (0x9dee bytes) 000710 1787788277.218050 write 0x0180a3 thru 0x027450 (0xf3ae bytes) 000711 1787788277.219258 mapread 0x00681b thru 0x008a66 (0x224c bytes) 000713 1787788277.220496 mapwrite 0x037653 thru 0x03ffff (0x89ad bytes) 000715 1787788277.286578 mapread 0x001d35 thru 0x00dd87 (0xc053 bytes) 000716 1787788277.287577 trunc from 0x040000 to 0x0211ae (0x1ee51 bytes) 000718 1787788277.303113 mapread 0x00cc05 thru 0x00e20b (0x1607 bytes) 000719 1787788277.308465 trunc from 0x0211af to 0x03b027 (0x19e79 bytes) 000720 1787788277.321606 mapwrite 0x01dc83 thru 0x026238 (0x85b6 bytes) 000721 1787788277.399348 write 0x015a31 thru 0x025287 (0xf857 bytes) 000723 1787788277.402234 write 0x032d40 thru 0x0389ab (0x5c6c bytes) 000724 1787788277.419818 mapread 0x02d64e thru 0x036a04 (0x93b7 bytes) 000725 1787788277.447590 trunc from 0x03b028 to 0x025846 (0x157e1 bytes) 000726 1787788277.474312 mapread 0x00bcc4 thru 0x011c06 (0x5f43 bytes) 000727 1787788277.475298 write 0x009cb4 thru 0x013a29 (0x9d76 bytes) 000728 1787788277.482679 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x25847): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25847): Mirror 2: 0x48f3e3d9 (stale) Mirror 3: 0x48f3e3d9 (stale) Mirror 4: 0x48f3e3d9 (stale) Mirror 5: 0x48f3e3d9 (stale) CRC-32 checksum value for chunk [0, 0x25847): Mirror 1: 0x8dc23924 Mirror 6: 0x8dc23924 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25847: skipped 000729 1787788278.656672 trunc from 0x025847 to 0x02010e (0x5738 bytes) 000730 1787788278.729514 trunc from 0x02010f to 0x030cb2 (0x10ba4 bytes) 000732 1787788278.747470 trunc from 0x030cb3 to 0x00d92b (0x23387 bytes) 000733 1787788278.764500 mapread 0x001046 thru 0x003dba (0x2d75 bytes) 000734 1787788278.787198 mapwrite 0x03a108 thru 0x03ffff (0x5ef8 bytes) 000735 1787788278.850892 trunc from 0x040000 to 0x01ee7f (0x21180 bytes) 000736 1787788278.865945 mapwrite 0x02e274 thru 0x031d43 (0x3ad0 bytes) 000740 1787788278.927105 trunc from 0x031d44 to 0x0317eb (0x558 bytes) 000741 1787788278.942018 mapwrite 0x00fd2e thru 0x01ef4e (0xf221 bytes) 000744 1787788279.001609 mapread 0x013239 thru 0x01beb9 (0x8c81 bytes) 000746 1787788279.004067 trunc from 0x0317ec to 0x038eef (0x7704 bytes) 000747 1787788279.021182 read 0x01f944 thru 0x022fec (0x36a9 bytes) 000751 1787788279.027441 mapwrite 0x0194f9 thru 0x01fa28 (0x6530 bytes) 000752 1787788279.071813 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ef0): Mirror 3: 0xcd42645f (stale) Mirror 4: 0xcd42645f (stale) Mirror 5: 0xcd42645f (stale) Mirror 6: 0xf7224795 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ef0): Mirror 3: 0xcd42645f (stale) Mirror 4: 0xcd42645f (stale) Mirror 5: 0xcd42645f (stale) Mirror 6: 0xf7224795 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ef0: skipped 000753 1787788279.391765 mapread 0x01555a thru 0x018672 (0x3119 bytes) 000754 1787788279.413333 mapread 0x026fa8 thru 0x03427a (0xd2d3 bytes) 000756 1787788279.430561 mapwrite 0x0331d6 thru 0x03ffff (0xce2a bytes) 000758 1787788279.512253 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4aea2943 (stale) Mirror 4: 0x4aea2943 (stale) Mirror 5: 0x4aea2943 (stale) Mirror 6: 0xf64faba0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7805a6b6 Mirror 7: 0x7805a6b6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000759 1787788280.244979 write 0x033d57 thru 0x0390d6 (0x5380 bytes) 000761 1787788280.316048 mapwrite 0x011104 thru 0x011670 (0x56d bytes) 000766 1787788280.396075 mapwrite 0x025d64 thru 0x032696 (0xc933 bytes) 000767 1787788280.461470 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4aea2943 (stale) Mirror 4: 0x4aea2943 (stale) Mirror 5: 0x4aea2943 (stale) Mirror 6: 0xf64faba0 (stale) Mirror 7: 0x7805a6b6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x54cc03c1 Mirror 8: 0x54cc03c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000771 1787788281.176980 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4aea2943 (stale) Mirror 4: 0x4aea2943 (stale) Mirror 5: 0x4aea2943 (stale) Mirror 7: 0x7805a6b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4aea2943 (stale) Mirror 4: 0x4aea2943 (stale) Mirror 5: 0x4aea2943 (stale) Mirror 7: 0x7805a6b6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x54cc03c1 Mirror 8: 0x54cc03c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000772 1787788281.640681 mapread 0x03256d thru 0x0370f4 (0x4b88 bytes) 000774 1787788281.672505 mapwrite 0x0162df thru 0x0174b9 (0x11db bytes) 000776 1787788281.868663 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4aea2943 (stale) Mirror 4: 0x4aea2943 (stale) Mirror 7: 0x7805a6b6 (stale) Mirror 8: 0x54cc03c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4aea2943 (stale) Mirror 4: 0x4aea2943 (stale) Mirror 7: 0x7805a6b6 (stale) Mirror 8: 0x54cc03c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000778 1787788282.322236 mapwrite 0x005bc6 thru 0x00ae24 (0x525f bytes) 000780 1787788282.414070 MIRROR_RESYNC 5 mirrors 000781 1787788283.254233 write 0x00f820 thru 0x01dbb8 (0xe399 bytes) 000782 1787788283.345705 write 0x01d0bb thru 0x0291c8 (0xc10e bytes) 000784 1787788283.352414 write 0x0127ee thru 0x012ecf (0x6e2 bytes) 000785 1787788283.353111 trunc from 0x040000 to 0x024f00 (0x1b0ff bytes) 000786 1787788283.406485 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24f01): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24f01): Mirror 3: 0xea42e87f (stale) Mirror 4: 0xea42e87f (stale) Mirror 7: 0xea42e87f (stale) Mirror 8: 0xea42e87f (stale) CRC-32 checksum value for chunk [0, 0x24f01): Mirror 1: 0xfe25b6a5 Mirror 9: 0xfe25b6a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24f01: skipped 000787 1787788284.178214 MIRROR_RESYNC 6 mirrors 000788 1787788285.189188 mapwrite 0x025bc8 thru 0x02b0a9 (0x54e2 bytes) 000791 1787788285.376182 MIRROR_RESYNC 6 mirrors 000793 1787788286.377634 trunc from 0x02b0aa to 0x033053 (0x7faa bytes) 000794 1787788286.444711 trunc from 0x033054 to 0x02f0a5 (0x3fae bytes) 000795 1787788286.469055 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2f0a6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f0a6): Mirror 3: 0x3a9789c4 (stale) Mirror 4: 0x3a9789c4 (stale) Mirror 7: 0x3a9789c4 (stale) Mirror 8: 0x3a9789c4 (stale) Mirror 9: 0x3a9789c4 (stale) CRC-32 checksum value for chunk [0, 0x2f0a6): Mirror 1: 0x3a9789c4 Mirror 10: 0x3a9789c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f0a6: skipped 000796 1787788287.353488 mapwrite 0x00ac3a thru 0x017bbe (0xcf85 bytes) 000797 1787788287.562823 MIRROR_RESYNC 7 mirrors 000798 1787788288.586670 mapread 0x01c3d3 thru 0x027d62 (0xb990 bytes) 000799 1787788288.611701 write 0x02a964 thru 0x02d359 (0x29f6 bytes) 000801 1787788288.678604 read 0x02cf2d thru 0x02f0a5 (0x2179 bytes) 000802 1787788288.684892 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2f0a6): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f0a6): Mirror 3: 0xd884905 (stale) Mirror 4: 0xd884905 (stale) Mirror 7: 0xd884905 (stale) Mirror 8: 0xd884905 (stale) Mirror 9: 0xd884905 (stale) Mirror 10: 0xd884905 (stale) CRC-32 checksum value for chunk [0, 0x2f0a6): Mirror 1: 0x8dd6858c Mirror 11: 0x8dd6858c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f0a6: skipped 000804 1787788289.469425 write 0x02cf6e thru 0x02e00d (0x10a0 bytes) 000808 1787788289.595842 mapread 0x028b31 thru 0x02f0a5 (0x6575 bytes) 000809 1787788289.617098 trunc from 0x02f0a6 to 0x036fe1 (0x7f3c bytes) 000810 1787788289.686040 read 0x02ebf3 thru 0x036fe1 (0x83ef bytes) 000811 1787788289.712488 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x36fe2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36fe2): Mirror 3: 0xdffc9e56 (stale) Mirror 4: 0xdffc9e56 (stale) Mirror 7: 0xdffc9e56 (stale) Mirror 8: 0xdffc9e56 (stale) Mirror 9: 0xdffc9e56 (stale) Mirror 10: 0xdffc9e56 (stale) Mirror 11: 0x8b4f6fcf (stale) CRC-32 checksum value for chunk [0, 0x36fe2): Mirror 1: 0xfa133937 Mirror 12: 0xfa133937 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36fe2: skipped 000815 1787788290.790895 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x36fe2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36fe2): Mirror 3: 0xdffc9e56 (stale) Mirror 4: 0xdffc9e56 (stale) Mirror 7: 0xdffc9e56 (stale) Mirror 8: 0xdffc9e56 (stale) Mirror 9: 0xdffc9e56 (stale) Mirror 10: 0xdffc9e56 (stale) Mirror 11: 0x8b4f6fcf (stale) CRC-32 checksum value for chunk [0, 0x36fe2): Mirror 1: 0xfa133937 Mirror 12: 0xfa133937 Mirror 13: 0xfa133937 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36fe2: skipped 000816 1787788291.921553 read 0x034b53 thru 0x036fe1 (0x248f bytes) 000818 1787788291.928082 mapread 0x0335ba thru 0x036fe1 (0x3a28 bytes) 000819 1787788291.933295 read 0x00648f thru 0x0157b5 (0xf327 bytes) 000820 1787788291.942345 read 0x01da60 thru 0x021694 (0x3c35 bytes) 000821 1787788291.946263 MIRROR_RESYNC 10 mirrors 000824 1787788293.727310 read 0x01f031 thru 0x028d62 (0x9d32 bytes) 000826 1787788293.738052 MIRROR_EXTEND to 11 mirrors 000827 1787788294.579792 write 0x013635 thru 0x01d63c (0xa008 bytes) 000828 1787788294.665558 trunc from 0x036fe2 to 0x0123b7 (0x24c2a bytes) 000830 1787788294.693247 trunc from 0x0123b8 to 0x001db3 (0x10604 bytes) 000832 1787788294.711777 trunc from 0x001db4 to 0x01be07 (0x1a054 bytes) 000834 1787788294.745201 mapread 0x01a6ec thru 0x01be07 (0x171c bytes) 000836 1787788294.763575 trunc from 0x01be08 to 0x03c631 (0x2082a bytes) 000837 1787788294.787132 mapwrite 0x036a3f thru 0x0376a0 (0xc62 bytes) 000838 1787788294.897602 mapwrite 0x03cc2e thru 0x03ffff (0x33d2 bytes) 000840 1787788294.996520 write 0x0085ef thru 0x0116b8 (0x90ca bytes) 000841 1787788295.010807 read 0x03b082 thru 0x03ed10 (0x3c8f bytes) 000843 1787788295.015473 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xde36be53 (stale) Mirror 4: 0xde36be53 (stale) Mirror 7: 0xde36be53 (stale) Mirror 8: 0xde36be53 (stale) Mirror 9: 0xde36be53 (stale) Mirror 10: 0xde36be53 (stale) Mirror 11: 0xde36be53 (stale) Mirror 12: 0xde36be53 (stale) Mirror 13: 0xde36be53 (stale) Mirror 14: 0xde36be53 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xda3e9abb Mirror 15: 0xda3e9abb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000845 1787788295.957786 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xde36be53 (stale) Mirror 4: 0xde36be53 (stale) Mirror 7: 0xde36be53 (stale) Mirror 8: 0xde36be53 (stale) Mirror 9: 0xde36be53 (stale) Mirror 10: 0xde36be53 (stale) Mirror 11: 0xde36be53 (stale) Mirror 12: 0xde36be53 (stale) Mirror 13: 0xde36be53 (stale) Mirror 14: 0xde36be53 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xda3e9abb Mirror 15: 0xda3e9abb Mirror 16: 0xda3e9abb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x3fddb 000846 1787788296.947322 trunc from 0x040000 to 0x03fdda (0x225 bytes) 000849 1787788297.025675 read 0x030804 thru 0x031e66 (0x1663 bytes) 000850 1787788297.032147 read 0x0046f7 thru 0x00c32b (0x7c35 bytes) 000852 1787788297.044252 trunc from 0x03fddb to 0x00fc2f (0x301ab bytes) 000853 1787788297.073499 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0xfc30): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfc30): Mirror 3: 0xeb74115b (stale) Mirror 4: 0xeb74115b (stale) Mirror 7: 0xeb74115b (stale) Mirror 8: 0xeb74115b (stale) Mirror 9: 0xeb74115b (stale) Mirror 10: 0xeb74115b (stale) Mirror 11: 0xeb74115b (stale) Mirror 12: 0xeb74115b (stale) Mirror 13: 0xeb74115b (stale) Mirror 14: 0xeb74115b (stale) Mirror 15: 0xfc54e682 (stale) Mirror 16: 0xfc54e682 (stale) CRC-32 checksum value for chunk [0, 0xfc30): Mirror 1: 0xfc54e682 Mirror 17: 0xfc54e682 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfc30: skipped 000854 1787788298.139217 mapwrite 0x027d42 thru 0x02eac1 (0x6d80 bytes) 000855 1787788298.409095 trunc from 0x02eac2 to 0x009765 (0x2535c bytes) 000856 1787788298.461859 MIRROR_RESYNC 14 mirrors 000857 1787788301.543505 mapread 0x006cf2 thru 0x009765 (0x2a74 bytes) 000858 1787788301.561987 write 0x01dd7a thru 0x02593c (0x7bc3 bytes) 000859 1787788301.645893 trunc from 0x02593d to 0x009ea3 (0x1ba99 bytes) 000860 1787788301.664241 read 0x0031b1 thru 0x009ea3 (0x6cf3 bytes) 000861 1787788301.675500 trunc from 0x009ea4 to 0x03fd0d (0x35e6a bytes) 000862 1787788301.692815 mapwrite 0x01c55c thru 0x024e81 (0x8926 bytes) 000867 1787788301.770104 mapread 0x027add thru 0x02af64 (0x3488 bytes) 000868 1787788301.785155 trunc from 0x03fd0e to 0x02b608 (0x14705 bytes) 000875 1787788301.810900 trunc from 0x02b609 to 0x036216 (0xac0e bytes) 000876 1787788301.832103 mapwrite 0x01b451 thru 0x01e0c5 (0x2c75 bytes) 000877 1787788301.871573 trunc from 0x036217 to 0x004f26 (0x312f0 bytes) 000878 1787788301.903379 trunc from 0x004f27 to 0x0380be (0x33198 bytes) 000879 1787788301.957009 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x380bf): Mirror 4: 0xd5a53fe3 (stale) Mirror 7: 0xd5a53fe3 (stale) Mirror 8: 0xd5a53fe3 (stale) Mirror 9: 0xd5a53fe3 (stale) Mirror 10: 0xd5a53fe3 (stale) Mirror 11: 0xd5a53fe3 (stale) Mirror 12: 0xd5a53fe3 (stale) Mirror 13: 0xd5a53fe3 (stale) Mirror 14: 0xd5a53fe3 (stale) Mirror 15: 0xd5a53fe3 (stale) Mirror 16: 0xd5a53fe3 (stale) Mirror 17: 0xd5a53fe3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380bf): Mirror 4: 0xd5a53fe3 (stale) Mirror 7: 0xd5a53fe3 (stale) Mirror 8: 0xd5a53fe3 (stale) Mirror 9: 0xd5a53fe3 (stale) Mirror 10: 0xd5a53fe3 (stale) Mirror 11: 0xd5a53fe3 (stale) Mirror 12: 0xd5a53fe3 (stale) Mirror 13: 0xd5a53fe3 (stale) Mirror 14: 0xd5a53fe3 (stale) Mirror 15: 0xd5a53fe3 (stale) Mirror 16: 0xd5a53fe3 (stale) Mirror 17: 0xd5a53fe3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380bf: skipped 000880 1787788302.744805 mapwrite 0x01e0b5 thru 0x0265fa (0x8546 bytes) 000881 1787788302.822139 mapread 0x00a163 thru 0x015982 (0xb820 bytes) 000882 1787788302.841145 write 0x00379f thru 0x008141 (0x49a3 bytes) 000883 1787788302.845968 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x380bf): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380bf): Mirror 4: 0xd5a53fe3 (stale) Mirror 7: 0xd5a53fe3 (stale) Mirror 8: 0xd5a53fe3 (stale) Mirror 9: 0xd5a53fe3 (stale) Mirror 10: 0xd5a53fe3 (stale) Mirror 11: 0xd5a53fe3 (stale) Mirror 12: 0xd5a53fe3 (stale) Mirror 13: 0xd5a53fe3 (stale) Mirror 14: 0xd5a53fe3 (stale) Mirror 15: 0xd5a53fe3 (stale) Mirror 16: 0xd5a53fe3 (stale) Mirror 17: 0xd5a53fe3 (stale) CRC-32 checksum value for chunk [0, 0x380bf): Mirror 1: 0x267b93a4 Mirror 18: 0x267b93a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380bf: skipped 000885 1787788304.276821 write 0x013d5d thru 0x01fec5 (0xc169 bytes) 000886 1787788304.369083 mapwrite 0x01b37d thru 0x0260a6 (0xad2a bytes) 000887 1787788304.438692 mapwrite 0x02aafb thru 0x02bd69 (0x126f bytes) 000889 1787788304.480501 mapwrite 0x01fefa thru 0x02b627 (0xb72e bytes) 000890 1787788304.523670 trunc from 0x0380bf to 0x03a589 (0x24cb bytes) 000891 1787788304.546222 trunc from 0x03a58a to 0x0345b8 (0x5fd1 bytes) 000892 1787788304.564497 trunc from 0x0345b9 to 0x0159f7 (0x1ebc1 bytes) 000893 1787788304.579847 read 0x0099b6 thru 0x0159f7 (0xc042 bytes) 000894 1787788304.587971 read 0x00d8c1 thru 0x0159f7 (0x8137 bytes) 000896 1787788304.589119 trunc from 0x0159f8 to 0x009fab (0xba4c bytes) 000899 1787788304.613115 MIRROR_RESYNC 14 mirrors 000901 1787788306.289916 read 0x003079 thru 0x009fab (0x6f33 bytes) 000902 1787788306.299181 mapwrite 0x00988b thru 0x00ce79 (0x35ef bytes) 000903 1787788306.465062 mapread 0x00887c thru 0x00ce79 (0x45fe bytes) 000906 1787788306.466922 read 0x0067be thru 0x0082d2 (0x1b15 bytes) 000909 1787788306.470930 trunc from 0x00ce7a to 0x0259b2 (0x18b39 bytes) 000910 1787788306.495633 read 0x0123af thru 0x01b4c2 (0x9114 bytes) 000911 1787788306.506142 mapread 0x00aac2 thru 0x013c2a (0x9169 bytes) 000912 1787788306.512265 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x259b3): Mirror 7: 0x4fc1c0cf (stale) Mirror 8: 0x4fc1c0cf (stale) Mirror 9: 0x4fc1c0cf (stale) Mirror 10: 0x4fc1c0cf (stale) Mirror 11: 0x4fc1c0cf (stale) Mirror 12: 0x4fc1c0cf (stale) Mirror 13: 0x4fc1c0cf (stale) Mirror 14: 0x4fc1c0cf (stale) Mirror 15: 0x4fc1c0cf (stale) Mirror 16: 0x4fc1c0cf (stale) Mirror 17: 0x4fc1c0cf (stale) Mirror 18: 0x4fc1c0cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x259b3): Mirror 7: 0x4fc1c0cf (stale) Mirror 8: 0x4fc1c0cf (stale) Mirror 9: 0x4fc1c0cf (stale) Mirror 10: 0x4fc1c0cf (stale) Mirror 11: 0x4fc1c0cf (stale) Mirror 12: 0x4fc1c0cf (stale) Mirror 13: 0x4fc1c0cf (stale) Mirror 14: 0x4fc1c0cf (stale) Mirror 15: 0x4fc1c0cf (stale) Mirror 16: 0x4fc1c0cf (stale) Mirror 17: 0x4fc1c0cf (stale) Mirror 18: 0x4fc1c0cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x259b3: skipped 000913 1787788307.063661 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x259b3): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x259b3): Mirror 7: 0x4fc1c0cf (stale) Mirror 8: 0x4fc1c0cf (stale) Mirror 9: 0x4fc1c0cf (stale) Mirror 10: 0x4fc1c0cf (stale) Mirror 11: 0x4fc1c0cf (stale) Mirror 12: 0x4fc1c0cf (stale) Mirror 13: 0x4fc1c0cf (stale) Mirror 14: 0x4fc1c0cf (stale) Mirror 15: 0x4fc1c0cf (stale) Mirror 16: 0x4fc1c0cf (stale) Mirror 17: 0x4fc1c0cf (stale) Mirror 18: 0x4fc1c0cf (stale) CRC-32 checksum value for chunk [0, 0x259b3): Mirror 1: 0xeb8b1fa6 Mirror 19: 0xeb8b1fa6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x259b3: skipped 000914 1787788308.314239 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x259b3): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x259b3): Mirror 7: 0x4fc1c0cf (stale) Mirror 8: 0x4fc1c0cf (stale) Mirror 9: 0x4fc1c0cf (stale) Mirror 10: 0x4fc1c0cf (stale) Mirror 11: 0x4fc1c0cf (stale) Mirror 12: 0x4fc1c0cf (stale) Mirror 13: 0x4fc1c0cf (stale) Mirror 14: 0x4fc1c0cf (stale) Mirror 15: 0x4fc1c0cf (stale) Mirror 16: 0x4fc1c0cf (stale) Mirror 17: 0x4fc1c0cf (stale) Mirror 18: 0x4fc1c0cf (stale) CRC-32 checksum value for chunk [0, 0x259b3): Mirror 1: 0xeb8b1fa6 Mirror 19: 0xeb8b1fa6 Mirror 20: 0xeb8b1fa6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x259b3: skipped 000915 1787788309.389338 mapwrite 0x0359cc thru 0x03ffff (0xa634 bytes) 000916 1787788309.573675 mapread 0x01decb thru 0x022895 (0x49cb bytes) 000917 1787788309.584114 write 0x03ca28 thru 0x03ffff (0x35d8 bytes) 000919 1787788309.590104 MIRROR_RESYNC 15 mirrors 000920 1787788313.147937 mapread 0x01e6fa thru 0x021d03 (0x360a bytes) 000921 1787788313.165112 mapwrite 0x0385e5 thru 0x03ffff (0x7a1b bytes) 000922 1787788313.302067 trunc from 0x040000 to 0x037afa (0x8505 bytes) 000925 1787788313.324987 mapread 0x03248b thru 0x037afa (0x5670 bytes) 000928 1787788313.327890 write 0x01602e thru 0x01e6b8 (0x868b bytes) 000930 1787788313.337786 write 0x002b3d thru 0x00fcfe (0xd1c2 bytes) 000931 1787788313.350237 MIRROR_RESYNC 15 mirrors 000933 1787788316.757892 mapread 0x0310f9 thru 0x037afa (0x6a02 bytes) 000934 1787788316.773092 write 0x033aa9 thru 0x03c120 (0x8678 bytes) 000937 1787788316.864065 mapwrite 0x011843 thru 0x01c70a (0xaec8 bytes) 000938 1787788316.942340 read 0x01a836 thru 0x01baf1 (0x12bc bytes) 000939 1787788316.942963 mapread 0x0074d3 thru 0x01285e (0xb38c bytes) 000942 1787788316.955529 write 0x01f6d7 thru 0x02c1a8 (0xcad2 bytes) 000944 1787788316.975638 MIRROR_RESYNC 15 mirrors 000945 1787788318.990151 MIRROR_RESYNC 15 mirrors 000947 1787788319.692247 mapwrite 0x0020c7 thru 0x009ba1 (0x7adb bytes) 000948 1787788319.866962 write 0x03e20d thru 0x03ffff (0x1df3 bytes) 000950 1787788319.869064 trunc from 0x040000 to 0x01d6ca (0x22935 bytes) 000951 1787788319.895257 MIRROR_SPLIT mirror 7 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d6cb): Mirror 7: 0xf9515339 (stale) Mirror 8: 0xf9515339 (stale) Mirror 9: 0xf9515339 (stale) Mirror 10: 0xf9515339 (stale) Mirror 11: 0xf9515339 (stale) Mirror 12: 0xf9515339 (stale) Mirror 14: 0xf9515339 (stale) Mirror 15: 0xf9515339 (stale) Mirror 16: 0xf9515339 (stale) Mirror 17: 0xf9515339 (stale) Mirror 18: 0xf9515339 (stale) Mirror 19: 0xf9515339 (stale) Mirror 20: 0xf9515339 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d6cb): Mirror 7: 0xf9515339 (stale) Mirror 8: 0xf9515339 (stale) Mirror 9: 0xf9515339 (stale) Mirror 10: 0xf9515339 (stale) Mirror 11: 0xf9515339 (stale) Mirror 12: 0xf9515339 (stale) Mirror 14: 0xf9515339 (stale) Mirror 15: 0xf9515339 (stale) Mirror 16: 0xf9515339 (stale) Mirror 17: 0xf9515339 (stale) Mirror 18: 0xf9515339 (stale) Mirror 19: 0xf9515339 (stale) Mirror 20: 0xf9515339 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d6cb: skipped 000953 1787788320.436500 trunc from 0x01d6cb to 0x0136e8 (0x9fe2 bytes) 000955 1787788320.457729 write 0x03f6d6 thru 0x03ffff (0x92a bytes) 000956 1787788320.460000 trunc from 0x040000 to 0x027dfe (0x18201 bytes) 000959 1787788320.486644 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x27dff): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27dff): Mirror 7: 0xc7aa06ef (stale) Mirror 8: 0xc7aa06ef (stale) Mirror 9: 0xc7aa06ef (stale) Mirror 10: 0xc7aa06ef (stale) Mirror 11: 0xc7aa06ef (stale) Mirror 12: 0xc7aa06ef (stale) Mirror 14: 0xc7aa06ef (stale) Mirror 15: 0xc7aa06ef (stale) Mirror 16: 0xc7aa06ef (stale) Mirror 17: 0xc7aa06ef (stale) Mirror 18: 0xc7aa06ef (stale) Mirror 19: 0xc7aa06ef (stale) Mirror 20: 0xc7aa06ef (stale) CRC-32 checksum value for chunk [0, 0x27dff): Mirror 1: 0x9a3d6e63 Mirror 21: 0x9a3d6e63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27dff: skipped 000961 1787788321.560562 write 0x03baff thru 0x03ffff (0x4501 bytes) 000962 1787788321.632350 trunc from 0x040000 to 0x03b211 (0x4dee bytes) 000963 1787788321.656584 trunc from 0x03b212 to 0x004626 (0x36beb bytes) 000965 1787788321.686362 mapwrite 0x0271e2 thru 0x02e4b5 (0x72d4 bytes) 000968 1787788321.867219 read 0x0136fd thru 0x016b2b (0x342f bytes) 000970 1787788321.875590 write 0x00013a thru 0x0022f1 (0x21b8 bytes) 000973 1787788321.885310 trunc from 0x02e4b6 to 0x03e1e2 (0xfd2d bytes) 000975 1787788321.897096 read 0x00b7a8 thru 0x019cd9 (0xe532 bytes) 000977 1787788321.903953 read 0x0187be thru 0x020419 (0x7c5c bytes) 000978 1787788321.904533 mapwrite 0x004cc6 thru 0x0102bf (0xb5fa bytes) 000979 1787788321.931723 read 0x014837 thru 0x020006 (0xb7d0 bytes) 000982 1787788321.933587 mapwrite 0x0097b4 thru 0x01249b (0x8ce8 bytes) 000983 1787788321.977718 mapwrite 0x000421 thru 0x005fbe (0x5b9e bytes) 000984 1787788322.071166 read 0x0153f7 thru 0x02163f (0xc249 bytes) 000985 1787788322.079815 MIRROR_RESYNC 15 mirrors 000988 1787788324.615941 mapwrite 0x01dee6 thru 0x0265c3 (0x86de bytes) 000990 1787788324.840468 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3e1e3): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1e3): Mirror 7: 0xf6033b5f (stale) Mirror 8: 0xf6033b5f (stale) Mirror 9: 0xf6033b5f (stale) Mirror 10: 0xf6033b5f (stale) Mirror 11: 0xf6033b5f (stale) Mirror 12: 0xf6033b5f (stale) Mirror 14: 0xf6033b5f (stale) Mirror 15: 0xf6033b5f (stale) Mirror 16: 0xf6033b5f (stale) Mirror 17: 0xf6033b5f (stale) Mirror 18: 0xf6033b5f (stale) Mirror 19: 0xf6033b5f (stale) Mirror 20: 0xf6033b5f (stale) Mirror 21: 0xf6033b5f (stale) CRC-32 checksum value for chunk [0, 0x3e1e3): Mirror 1: 0x3df699ce Mirror 22: 0x3df699ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1e3: skipped 000991 1787788325.873637 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a7:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e1e3): Mirror 8: 0xf6033b5f (stale) Mirror 9: 0xf6033b5f (stale) Mirror 10: 0xf6033b5f (stale) Mirror 11: 0xf6033b5f (stale) Mirror 12: 0xf6033b5f (stale) Mirror 14: 0xf6033b5f (stale) Mirror 15: 0xf6033b5f (stale) Mirror 16: 0xf6033b5f (stale) Mirror 17: 0xf6033b5f (stale) Mirror 18: 0xf6033b5f (stale) Mirror 19: 0xf6033b5f (stale) Mirror 20: 0xf6033b5f (stale) Mirror 21: 0xf6033b5f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1e3): Mirror 8: 0xf6033b5f (stale) Mirror 9: 0xf6033b5f (stale) Mirror 10: 0xf6033b5f (stale) Mirror 11: 0xf6033b5f (stale) Mirror 12: 0xf6033b5f (stale) Mirror 14: 0xf6033b5f (stale) Mirror 15: 0xf6033b5f (stale) Mirror 16: 0xf6033b5f (stale) Mirror 17: 0xf6033b5f (stale) Mirror 18: 0xf6033b5f (stale) Mirror 19: 0xf6033b5f (stale) Mirror 20: 0xf6033b5f (stale) Mirror 21: 0xf6033b5f (stale) CRC-32 checksum value for chunk [0, 0x3e1e3): Mirror 1: 0x3df699ce Mirror 22: 0x3df699ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1e3: skipped 000992 1787788326.832668 write 0x0036e1 thru 0x004868 (0x1188 bytes) 000993 1787788326.931274 read 0x02a3df thru 0x035056 (0xac78 bytes) 000996 1787788326.938313 read 0x027b73 thru 0x02f66c (0x7afa bytes) 000997 1787788326.948152 read 0x00f2ac thru 0x019644 (0xa399 bytes) 000998 1787788326.948864 write 0x012ee9 thru 0x01d5b2 (0xa6ca bytes) All operations completed A-OK! PASS 70a (103s) == sanity-flr test 71: check mirror extend parent fid ==== 19:52:12 (1787788332) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 19:52:16 (1787788336) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18824 s, 11.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163475 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x300:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ff:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e5:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 19:52:24 (1787788344) Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre2 Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 6000224 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6139744 .. failed Extending file size to 6865952 .. Extending file size to 8238976 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8591264 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8649088 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8763776 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8912000 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8856224 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58773 58775 58776 58779 58780 done 192.168.202.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg208-client.virtnet /mnt/lustre2 (opts:) 192.168.202.108@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg208-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:53:44 (1787788424) Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre2 Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4334400 .. Extending file size to 7099360 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6065216 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6498272 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7637344 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7541120 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7668896 .. Extending file size to 8071104 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8382464 .. Extending file size to 8100576 .. failed Extending file size to 9019712 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8395584 .. Extending file size to 8634208 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8760480 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 60307 60308 60309 60313 60314 done 192.168.202.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg208-client.virtnet /mnt/lustre2 (opts:) 192.168.202.108@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg208-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:55:05 (1787788505) Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145743 s, 14.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.444177 s, 4.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xf7f6868c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg208-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:55:26 (1787788526) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107925 s, 19.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 19:55:34 (1787788534) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fc:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x300:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:55:42 (1787788542) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158667 s, 13.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x301:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:55:50 (1787788550) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161985 s, 12.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:55:57 (1787788557) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167816 s, 12.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:56:30 (1787788590) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg208-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160545 s, 6.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:57:27 (1787788647) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg208-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.26808 s, 7.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:58:29 (1787788709) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg208-server mds-ost sync done. Creating new pool test_204f oleg208-server: Pool lustre.test_204f created Adding targets to pool oleg208-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg208-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168333 s, 12.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg208-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg208-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg208-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:59:44 (1787788784) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:59:51 (1787788791) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:59:59 (1787788799) oleg208-server: Pool lustre.test_206 created Adding targets to pool oleg208-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg208-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg208-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg208-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg208-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:00:23 (1787788823) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125763 s, 8.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572686 s, 18.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfda7f64b Mirror 3: 0xfda7f64b PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:00:33 (1787788833) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:00:36 (1787788836) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:00:40 (1787788840) Starting client: oleg208-client.virtnet: -o user_xattr,flock 192.168.202.108@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.108745 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x15c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23f0000 Read this (4 bytes): BBBB 192.168.202.108@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg208-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:01:17 (1787788877) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0235254 s, 348 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:01:23 (1787788883) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575116 s, 18.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x221 lmm_fid: [0x200000404:0x221:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x221 lmm_fid: [0x200000404:0x221:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x314:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834465 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 20:01:24.000000000 -0400 Modify: 2026-08-26 20:01:24.000000000 -0400 Change: 2026-08-26 20:01:26.000000000 -0400 Birth: 2026-08-26 20:01:23.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:01:31 (1787788891) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:01:35 (1787788895) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.615645 s, 66.5 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0483409 s, 84.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 20:01 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x223:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 20:01 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:01:45 (1787788905) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:01:53 (1787788913) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1789 sec =========== 20:02:03 (1787788923) === sanity-flr: start cleanup 20:02:05 (1787788925) === === sanity-flr: finish cleanup 20:02:11 (1787788931) ===