-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 19:20:14 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg215-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:20:35 (1787786435) === oleg215-client.virtnet: executing check_config_client /mnt/lustre oleg215-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg215-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a35e0639000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a35e0639000.idle_timeout=debug disable quota as required oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:21:11 (1787786471) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7641] == sanity-flr test 0a: lfs mirror create with -N option == 19:21:14 (1787786474) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:21:33 (1787786493) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:21:37 (1787786497) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:21:40 (1787786500) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:22:07 (1787786527) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:22:11 (1787786531) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 19:22:14 (1787786534) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8a35e0639000.stats=clear osc.lustre-OST0001-osc-ffff8a35e0639000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:22:23 (1787786543) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:22:34 (1787786554) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:22:44 (1787786564) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:22:49 (1787786569) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:22:59 (1787786579) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 19:23:10 (1787786590) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:23:18 (1787786598) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:23:21 (1787786601) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:23:30 (1787786610) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:23:42 (1787786622) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:23:50 (1787786630) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.446542 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111816 s, 9.4 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:24:01 (1787786641) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0517096 s, 40.6 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:24:11 (1787786651) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:24:23 (1787786663) Stopping /mnt/lustre-ost1 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a35e0639000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a35e0639000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg215-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg215-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a35e0639000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a35e0639000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg215-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (141s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:26:44 (1787786804) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189318 s, 11.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a35e0639000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a35e0639000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg215-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:27:33 (1787786853) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a35e0639000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a35e0639000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg215-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a35e0639000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a35e0639000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg215-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:29:05 (1787786945) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.14319 s, 10.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 19:29 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.518s user 0m0.001s sys 0m0.185s Stopping /mnt/lustre-ost1 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a35e0639000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a35e0639000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.619s user 0m0.001s sys 0m0.211s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg215-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg215-server oleg215-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a35e0639000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a35e0639000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.282s user 0m0.001s sys 0m0.211s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg215-server: oleg215-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg215-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg215-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a35e0639000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a35e0639000.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:30:46 (1787787046) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:30:50 (1787787050) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (5s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:30:55 (1787787055) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 19:30:59 (1787787059) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 19:31:07 (1787787067) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.640225 s, 19.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:31:22 (1787787082) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519754 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332888 s, 31.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:31:31 (1787787091) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:31:40 (1787787100) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102769 s, 996 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122397 s, 837 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00238149 s, 430 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00154583 s, 662 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 19:31:52 (1787787112) create test files with size 15743174 .. 0cfe5870f3f2da5fca5b13c3a6416309 - 16fb037e71e6e40b03f4d9840451713c - bad6e852f1e595e0d51ba0ad071b5381 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg215-client.virtnet /mnt/lustre (opts:) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 19:32:22 (1787787142) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (20s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:32:41 (1787787161) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:32:51 (1787787171) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133801 s, 7.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608087 s, 17.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 19:33:01 (1787787181) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199706 s, 10.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.387705 s, 10.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259495 s, 8.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 19:33:20 (1787787200) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 19:33:23 (1787787203) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:33:27 (1787787207) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267933 s, 39.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 19:33:35 (1787787215) llite.lustre-ffff8a35c3f64800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 19:33:43 (1787787223) llite.lustre-ffff8a35c3f64800.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278621 s, 37.6 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475603 s, 22.0 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 19:33:52 (1787787232) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:33:55 (1787787235) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xca:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xca:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:34:06 (1787787246) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.346579 s, 30.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg215-client.virtnet /mnt/lustre (opts:) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:34:18 (1787787258) llite.lustre-ffff8a35c2e99800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.312184 s, 33.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg215-client.virtnet /mnt/lustre (opts:) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8a35d30e2800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:34:33 (1787787273) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:35:17 (1787787317) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:35:54 (1787787354) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 19:35:57 (1787787357) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.208253 s, 15.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.167654 s, 18.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xac19298 (nosync,stale) Mirror 2: 0xac19298 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 27158adb0e619b5c13efbd107b38fea6 - mirror 1 checksum: 2d700673faeec6839d042aa6d45e0c26 - mirror 2 checksum: 27158adb0e619b5c13efbd107b38fea6 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T19:36:00-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T19:36:00-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 27158adb0e619b5c13efbd107b38fea6 - mirror 1 checksum: 2d700673faeec6839d042aa6d45e0c26 - mirror 2 checksum: 2d700673faeec6839d042aa6d45e0c26 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T19:36:00-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T19:36:00-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:36:10 (1787787370) Failing mds1 on oleg215-server Stopping /mnt/lustre-mds1 (opts:) on oleg215-server 19:36:18 (1787787378) shut down facet: mds1 facet_host: oleg215-server facet_failover_host: oleg215-server Failover mds1 to oleg215-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 19:36:54 (1787787414) targets are mounted 19:36:54 (1787787414) facet_failover done oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323233 s, 32.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe1:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:37:15 (1787787435) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:37:19 (1787787439) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:37:23 (1787787443) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0194403 s, 843 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:37:33 (1787787453) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:37:37 (1787787457) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0172129 s, 238 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.110296 s, 37.1 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:37:46 (1787787466) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:38:10 (1787787490) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:38:42 (1787787522) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:39:07 (1787787547) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:39:33 (1787787573) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 19:39:41 (1787787581) mirror create pid 55613 mirror split pid 55615 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55613 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 55615 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 19:40:49 (1787787649) Seed set to 7652 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x181d4 000005 1787787653.093772 trunc from 00000000 to 0x0181d3 (0x181d4 bytes) 000006 1787787653.121854 mapwrite 0x01829e thru 0x01b159 (0x2ebc bytes) 000011 1787787653.314734 read 0x01211e thru 0x01b159 (0x903c bytes) 000012 1787787653.344501 mapwrite 0x02af1a thru 0x039455 (0xe53c bytes) 000013 1787787653.519952 trunc from 0x039456 to 0x00aa02 (0x2ea53 bytes) 000014 1787787653.548804 read 0x004711 thru 0x0089e8 (0x42d8 bytes) 000015 1787787653.559409 mapread 0x007189 thru 0x0098d1 (0x2749 bytes) 000017 1787787653.562684 write 0x03c091 thru 0x03ffff (0x3f6f bytes) 000021 1787787653.566856 write 0x001c75 thru 0x010681 (0xea0d bytes) 000024 1787787653.577321 write 0x020417 thru 0x0295fd (0x91e7 bytes) 000025 1787787653.587533 mapwrite 0x004c1b thru 0x012278 (0xd65e bytes) 000026 1787787653.684785 write 0x0385bb thru 0x03ffff (0x7a45 bytes) 000027 1787787653.692267 write 0x00daf7 thru 0x00f90e (0x1e18 bytes) 000028 1787787653.703397 read 0x02a738 thru 0x02c536 (0x1dff bytes) 000030 1787787653.713844 mapwrite 0x026050 thru 0x032235 (0xc1e6 bytes) 000031 1787787653.924078 read 0x00938b thru 0x015d28 (0xc99e bytes) 000032 1787787653.924918 mapread 0x016281 thru 0x0208e4 (0xa664 bytes) 000034 1787787653.934372 read 0x013652 thru 0x01d90e (0xa2bd bytes) 000035 1787787653.935098 trunc from 0x040000 to 0x00de0c (0x321f3 bytes) 000036 1787787653.975352 write 0x02f446 thru 0x03b65b (0xc216 bytes) 000037 1787787653.995909 write 0x00409b thru 0x011916 (0xd87c bytes) 000041 1787787654.034481 trunc from 0x03b65c to 0x0129d1 (0x28c8a bytes) 000043 1787787654.132761 read 0x00e899 thru 0x0129d1 (0x4139 bytes) 000046 1787787654.175562 read 0x00582f thru 0x00d077 (0x7849 bytes) 000047 1787787654.176471 mapwrite 0x02054c thru 0x0249c9 (0x447e bytes) 000048 1787787654.458717 write 0x035946 thru 0x03ffff (0xa6ba bytes) 000049 1787787654.460283 write 0x0341bd thru 0x03ffff (0xbe43 bytes) truncating to largest ever: 0x3d565 000050 1787787654.482599 trunc from 0x040000 to 0x03d564 (0x2a9b bytes) 000051 1787787654.507625 read 0x037743 thru 0x03cb0a (0x53c8 bytes) 000052 1787787654.508322 write 0x002d75 thru 0x00be10 (0x909c bytes) 000053 1787787654.509546 mapwrite 0x0015e7 thru 0x001f53 (0x96d bytes) 000056 1787787654.680043 mapread 0x01c950 thru 0x022c1b (0x62cc bytes) 000057 1787787654.690175 mapread 0x035eac thru 0x03836e (0x24c3 bytes) 000059 1787787654.692424 mapwrite 0x03032c thru 0x03e522 (0xe1f7 bytes) 000060 1787787654.965001 mapwrite 0x0263db thru 0x0361f3 (0xfe19 bytes) 000062 1787787655.154238 mapread 0x01ba82 thru 0x0281a6 (0xc725 bytes) 000063 1787787655.155114 mapread 0x00c344 thru 0x014cd3 (0x8990 bytes) 000064 1787787655.167280 mapread 0x00185f thru 0x005075 (0x3817 bytes) 000066 1787787655.193382 MIRROR_EXTEND to 2 mirrors 000067 1787787657.545230 trunc from 0x03e523 to 0x0296b5 (0x14e6d bytes) 000068 1787787657.786188 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x296b6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x296b6): Mirror 2: 0xfcf4ece7 (stale) CRC-32 checksum value for chunk [0, 0x296b6): Mirror 1: 0xfcf4ece7 Mirror 3: 0xfcf4ece7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x296b6: skipped 000069 1787787660.086390 read 0x024ee0 thru 0x0296b5 (0x47d6 bytes) 000071 1787787660.099126 mapwrite 0x03afc6 thru 0x03ffff (0x503a bytes) 000072 1787787660.364797 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7865f282 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7865f282 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000073 1787787660.839090 mapwrite 0x0344c5 thru 0x03ca73 (0x85af bytes) 000075 1787787661.125603 read 0x01877e thru 0x01a575 (0x1df8 bytes) 000079 1787787661.142468 write 0x032744 thru 0x0355e5 (0x2ea2 bytes) 000080 1787787661.143876 mapwrite 0x017a3d thru 0x01e508 (0x6acc bytes) 000083 1787787661.249369 read 0x01f24f thru 0x02a92f (0xb6e1 bytes) 000084 1787787661.267273 read 0x02f276 thru 0x033e07 (0x4b92 bytes) 000085 1787787661.268105 write 0x01f4b6 thru 0x025677 (0x61c2 bytes) 000086 1787787661.280100 mapwrite 0x019da4 thru 0x01deba (0x4117 bytes) 000091 1787787661.351488 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7865f282 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8faef522 Mirror 3: 0x8faef522 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000092 1787787663.334868 trunc from 0x040000 to 0x0396a2 (0x695d bytes) 000095 1787787663.694611 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x396a3): Mirror 3: 0x200e3ff6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396a3): Mirror 3: 0x200e3ff6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396a3: skipped 000097 1787787664.340009 mapwrite 0x029f7f thru 0x02ef7d (0x4fff bytes) 000098 1787787664.586957 read 0x033526 thru 0x0396a2 (0x617d bytes) 000099 1787787664.648951 MIRROR_RESYNC 2 mirrors 000100 1787787665.964083 write 0x019485 thru 0x027fb8 (0xeb34 bytes) 000101 1787787666.120630 trunc from 0x0396a3 to 0x03be49 (0x27a7 bytes) 000102 1787787666.172623 trunc from 0x03be4a to 0x0063e3 (0x35a66 bytes) 000103 1787787666.243725 read 0x000c6b thru 0x0063e3 (0x5779 bytes) 000105 1787787666.276939 mapread 0x001b8e thru 0x0063e3 (0x4856 bytes) 000107 1787787666.292504 write 0x02f054 thru 0x034025 (0x4fd2 bytes) 000108 1787787666.304069 trunc from 0x034026 to 0x0342da (0x2b5 bytes) 000109 1787787666.359216 trunc from 0x0342db to 0x02276f (0x11b6b bytes) 000110 1787787666.396264 mapwrite 0x029279 thru 0x035f3c (0xccc4 bytes) 000111 1787787666.596618 MIRROR_RESYNC 2 mirrors 000112 1787787667.704077 trunc from 0x035f3d to 0x00c566 (0x299d6 bytes) 000113 1787787667.830763 trunc from 0x00c567 to 0x009f9f (0x25c7 bytes) 000114 1787787667.850039 mapread 0x009a89 thru 0x009f9f (0x517 bytes) 000116 1787787667.857439 write 0x0045bb thru 0x00f6a9 (0xb0ef bytes) 000118 1787787667.863102 trunc from 0x00f6aa to 0x00b24e (0x445b bytes) 000122 1787787667.889841 write 0x01d110 thru 0x027f27 (0xae18 bytes) 000123 1787787667.892897 read 0x00f455 thru 0x01ca20 (0xd5cc bytes) 000126 1787787667.908166 read 0x005af1 thru 0x00bec8 (0x63d8 bytes) 000130 1787787667.912013 mapwrite 0x019322 thru 0x019a8a (0x769 bytes) 000131 1787787667.975862 trunc from 0x027f28 to 0x018702 (0xf825 bytes) 000132 1787787668.002927 trunc from 0x018703 to 0x00563a (0x130c8 bytes) 000133 1787787668.032520 read 0x000827 thru 0x00563a (0x4e14 bytes) 000134 1787787668.049119 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000138 1787787668.235511 mapwrite 0x00d610 thru 0x01906d (0xba5e bytes) 000139 1787787668.507743 trunc from 0x01906e to 0x016713 (0x295a bytes) 000144 1787787668.561647 write 0x03509e thru 0x03f488 (0xa3eb bytes) 000148 1787787668.566722 write 0x02455f thru 0x031f6b (0xda0d bytes) 000150 1787787668.582713 trunc from 0x03f489 to 0x00bac9 (0x339bf bytes) 000153 1787787668.645943 trunc from 0x00baca to 0x02f5a2 (0x23ad9 bytes) 000154 1787787668.678236 trunc from 0x02f5a3 to 0x0062ff (0x292a3 bytes) 000156 1787787668.722094 read 0x000d55 thru 0x0062ff (0x55ab bytes) 000157 1787787668.730037 read 0x003d1b thru 0x0062ff (0x25e5 bytes) 000162 1787787668.737463 mapwrite 0x0215fb thru 0x029e17 (0x881d bytes) 000164 1787787668.873693 write 0x03395b thru 0x03ffff (0xc6a5 bytes) 000165 1787787668.879432 read 0x00c46a thru 0x01215b (0x5cf2 bytes) 000166 1787787668.885013 trunc from 0x040000 to 0x0196fb (0x26904 bytes) 000168 1787787668.916519 mapread 0x00f588 thru 0x0196fb (0xa174 bytes) 000173 1787787668.926347 mapwrite 0x015662 thru 0x0208e2 (0xb281 bytes) 000176 1787787669.045774 read 0x0191c7 thru 0x01c0f7 (0x2f31 bytes) 000177 1787787669.046397 read 0x0193eb thru 0x0208e2 (0x74f8 bytes) 000180 1787787669.049388 mapread 0x01cdff thru 0x0208e2 (0x3ae4 bytes) 000182 1787787669.066243 trunc from 0x0208e3 to 0x028d6c (0x848a bytes) 000183 1787787669.102802 trunc from 0x028d6d to 0x023cc2 (0x50aa bytes) 000184 1787787669.137009 trunc from 0x023cc3 to 0x03b63b (0x17979 bytes) 000185 1787787669.175747 mapread 0x014237 thru 0x014923 (0x6ed bytes) 000187 1787787669.177439 mapread 0x0320d8 thru 0x03844a (0x6373 bytes) 000189 1787787669.191849 trunc from 0x03b63c to 0x03bad6 (0x49b bytes) 000192 1787787669.226556 mapread 0x030044 thru 0x03237b (0x2338 bytes) 000193 1787787669.231379 read 0x00f4f9 thru 0x00f851 (0x359 bytes) 000195 1787787669.238537 mapread 0x013640 thru 0x015660 (0x2021 bytes) 000196 1787787669.274560 trunc from 0x03bad7 to 0x021ff2 (0x19ae4 bytes) 000197 1787787669.299650 mapread 0x0213fd thru 0x021ff2 (0xbf6 bytes) 000201 1787787669.304038 mapwrite 0x00017a thru 0x00a14f (0x9fd6 bytes) 000202 1787787669.427428 trunc from 0x021ff3 to 0x0105be (0x11a34 bytes) 000203 1787787669.449824 mapread 0x00b4ef thru 0x0105be (0x50d0 bytes) 000205 1787787669.454123 trunc from 0x0105bf to 0x0062f5 (0xa2c9 bytes) 000206 1787787669.490243 trunc from 0x0062f6 to 0x02009b (0x19da6 bytes) 000208 1787787669.563748 mapread 0x01117e thru 0x01207b (0xefe bytes) 000216 1787787669.598713 write 0x02eb61 thru 0x035e54 (0x72f4 bytes) 000217 1787787669.607734 mapread 0x01428f thru 0x021b8a (0xd8fc bytes) 000218 1787787669.633464 trunc from 0x035e55 to 0x000ad3 (0x35381 bytes) 000219 1787787669.677823 mapread 0x00031c thru 0x000ad3 (0x7b8 bytes) 000220 1787787669.680824 mapread 0x000a38 thru 0x000ad3 (0x9c bytes) 000221 1787787669.697003 MIRROR_EXTEND to 2 mirrors 000222 1787787671.031114 mapread 0x000251 thru 0x000891 (0x641 bytes) 000227 1787787671.047086 read 0x000417 thru 0x000ad3 (0x6bd bytes) 000228 1787787671.047928 mapread 0x00022a thru 0x000ad3 (0x8aa bytes) 000229 1787787671.049085 trunc from 0x000ad4 to 0x01ce09 (0x1c336 bytes) 000230 1787787671.127616 trunc from 0x01ce0a to 0x01a302 (0x2b07 bytes) 000231 1787787671.147202 mapread 0x000668 thru 0x00a4e7 (0x9e80 bytes) 000232 1787787671.169940 mapread 0x01739c thru 0x01a302 (0x2f67 bytes) 000233 1787787671.178632 mapread 0x00a848 thru 0x0101df (0x5998 bytes) 000236 1787787671.184421 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1a303): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a303): Mirror 2: 0x5ae8cb9a (stale) CRC-32 checksum value for chunk [0, 0x1a303): Mirror 1: 0x5ae8cb9a Mirror 3: 0x5ae8cb9a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a303: skipped 000237 1787787672.214754 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000238 1787787672.528552 mapwrite 0x03717a thru 0x038347 (0x11ce bytes) 000240 1787787672.680408 write 0x027171 thru 0x030113 (0x8fa3 bytes) 000243 1787787672.691379 mapread 0x019fec thru 0x024927 (0xa93c bytes) 000244 1787787672.708368 write 0x033365 thru 0x033e45 (0xae1 bytes) 000247 1787787672.720182 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38348): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38348): Mirror 3: 0x8cc38817 (stale) CRC-32 checksum value for chunk [0, 0x38348): Mirror 1: 0x63e40cc9 Mirror 4: 0x63e40cc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38348: skipped 000251 1787787673.739654 write 0x001071 thru 0x00a3db (0x936b bytes) 000252 1787787673.858915 read 0x02a7ee thru 0x035d75 (0xb588 bytes) 000253 1787787673.867433 trunc from 0x038348 to 0x03c0cf (0x3d88 bytes) 000257 1787787673.884929 trunc from 0x03c0d0 to 0x017559 (0x24b76 bytes) 000258 1787787673.914036 read 0x008dcc thru 0x01632a (0xd55f bytes) 000259 1787787673.940768 MIRROR_RESYNC 3 mirrors 000260 1787787674.935516 mapwrite 0x03e0e5 thru 0x03ffff (0x1f1b bytes) 000264 1787787675.095202 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x939fd09b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x939fd09b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000265 1787787675.391620 mapread 0x02ad8e thru 0x03373b (0x89ae bytes) 000266 1787787675.410690 read 0x0372a7 thru 0x03e07b (0x6dd5 bytes) 000270 1787787675.420320 mapread 0x029515 thru 0x02d221 (0x3d0d bytes) 000271 1787787675.421361 read 0x011781 thru 0x01667a (0x4efa bytes) 000272 1787787675.443721 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) 000273 1787787675.579612 mapwrite 0x000af7 thru 0x007d93 (0x729d bytes) 000275 1787787675.701996 mapwrite 0x02cafb thru 0x02f31e (0x2824 bytes) 000276 1787787675.786641 mapread 0x027daf thru 0x0355d9 (0xd82b bytes) 000278 1787787675.797722 trunc from 0x040000 to 0x03ce79 (0x3186 bytes) 000283 1787787675.828636 mapwrite 0x02bc78 thru 0x030c16 (0x4f9f bytes) 000284 1787787675.910042 read 0x008633 thru 0x016810 (0xe1de bytes) skipping zero size read 000287 1787787675.924396 trunc from 0x03ce7a to 0x01930a (0x23b6f bytes) 000288 1787787675.972006 write 0x03ce92 thru 0x03d650 (0x7bf bytes) 000289 1787787675.974530 mapread 0x0385d1 thru 0x039586 (0xfb6 bytes) 000295 1787787676.030987 mapwrite 0x033c3b thru 0x03ee6a (0xb230 bytes) 000296 1787787676.175930 mapread 0x0193e8 thru 0x027908 (0xe521 bytes) 000297 1787787676.218680 trunc from 0x03ee6b to 0x00f33a (0x2fb30 bytes) 000302 1787787676.278736 read 0x004c5b thru 0x006649 (0x19ef bytes) 000303 1787787676.279386 mapwrite 0x004363 thru 0x00e2cc (0x9f6a bytes) 000304 1787787676.369468 read 0x00dcdb thru 0x00f33a (0x1660 bytes) 000305 1787787676.370103 mapwrite 0x026b40 thru 0x02cbfd (0x60be bytes) 000308 1787787676.523678 trunc from 0x02cbfe to 0x01c2da (0x10923 bytes) 000309 1787787676.559690 mapwrite 0x0224ec thru 0x026757 (0x426c bytes) 000310 1787787676.767562 write 0x007442 thru 0x0098e8 (0x24a7 bytes) 000315 1787787676.780624 mapwrite 0x02b591 thru 0x03770d (0xc17d bytes) 000317 1787787676.979225 mapread 0x023b4d thru 0x02a58f (0x6a43 bytes) 000319 1787787677.032604 read 0x034ef1 thru 0x03770d (0x281d bytes) 000320 1787787677.033247 mapread 0x035290 thru 0x03770d (0x247e bytes) 000321 1787787677.034068 read 0x0184c2 thru 0x0260cf (0xdc0e bytes) 000323 1787787677.073190 mapread 0x01b547 thru 0x01ee1f (0x38d9 bytes) 000325 1787787677.082383 write 0x029489 thru 0x02cb9d (0x3715 bytes) 000326 1787787677.084700 trunc from 0x03770e to 0x016c68 (0x20aa5 bytes) 000327 1787787677.153008 mapread 0x003107 thru 0x004dc1 (0x1cbb bytes) 000328 1787787677.181304 MIRROR_EXTEND to 2 mirrors 000329 1787787678.279298 mapwrite 0x002601 thru 0x008c0b (0x660b bytes) 000330 1787787678.504616 write 0x029a4a thru 0x02bd41 (0x22f8 bytes) 000331 1787787678.506027 mapread 0x01adde thru 0x02822d (0xd450 bytes) 000332 1787787678.520777 mapread 0x018fca thru 0x01bfca (0x3001 bytes) 000334 1787787678.527033 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2bd42): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bd42): Mirror 2: 0x221fbb19 (stale) CRC-32 checksum value for chunk [0, 0x2bd42): Mirror 1: 0x84c5215a Mirror 3: 0x84c5215a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bd42: skipped 000336 1787787679.589202 mapwrite 0x036f63 thru 0x03dc00 (0x6c9e bytes) 000341 1787787679.844415 read 0x024997 thru 0x02acb3 (0x631d bytes) 000343 1787787679.857032 read 0x002320 thru 0x00b87e (0x955f bytes) 000344 1787787679.888081 trunc from 0x03dc01 to 0x012a53 (0x2b1ad bytes) 000345 1787787679.953810 MIRROR_RESYNC 3 mirrors 000347 1787787681.180651 write 0x02f32a thru 0x0360f9 (0x6dd0 bytes) 000348 1787787681.247439 read 0x0247f4 thru 0x02db0f (0x931c bytes) 000350 1787787681.256098 mapread 0x026fd6 thru 0x02c0cb (0x50f6 bytes) 000351 1787787681.257609 read 0x009d3a thru 0x013ea3 (0xa16a bytes) 000352 1787787681.271354 MIRROR_RESYNC 3 mirrors 000357 1787787682.473165 mapread 0x01ca08 thru 0x027479 (0xaa72 bytes) 000361 1787787682.529999 mapread 0x00a653 thru 0x017eb3 (0xd861 bytes) 000366 1787787682.593003 MIRROR_RESYNC 3 mirrors 000369 1787787683.103025 trunc from 0x0360fa to 0x02fbd0 (0x6529 bytes) 000371 1787787683.223755 mapread 0x01c84f thru 0x02c2c5 (0xfa77 bytes) 000372 1787787683.268424 write 0x01d903 thru 0x01de50 (0x54e bytes) 000373 1787787683.281325 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2fbd1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fbd1): Mirror 2: 0x6b4a1c (stale) Mirror 3: 0x6b4a1c (stale) CRC-32 checksum value for chunk [0, 0x2fbd1): Mirror 1: 0x180edba0 Mirror 4: 0x180edba0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fbd1: skipped 000377 1787787684.866866 mapwrite 0x031f57 thru 0x03386f (0x1919 bytes) 000378 1787787685.085705 trunc from 0x033870 to 0x028401 (0xb46e bytes) 000381 1787787685.114655 mapread 0x0037e9 thru 0x00594c (0x2164 bytes) 000383 1787787685.127358 mapread 0x0125db thru 0x015166 (0x2b8c bytes) 000385 1787787685.138685 mapread 0x0259b3 thru 0x028401 (0x2a4f bytes) 000386 1787787685.149444 read 0x025eb2 thru 0x028401 (0x2550 bytes) 000388 1787787685.150512 mapread 0x01fc0d thru 0x028401 (0x87f5 bytes) 000394 1787787685.160032 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x28402): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28402): Mirror 2: 0xd840137a (stale) Mirror 3: 0xd840137a (stale) Mirror 4: 0xd978de5d (stale) CRC-32 checksum value for chunk [0, 0x28402): Mirror 1: 0xd978de5d Mirror 5: 0xd978de5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28402: skipped 000395 1787787686.137497 write 0x0101b1 thru 0x012e00 (0x2c50 bytes) 000398 1787787686.231531 read 0x01dcca thru 0x028401 (0xa738 bytes) 000399 1787787686.239400 mapwrite 0x0374f9 thru 0x03b0a9 (0x3bb1 bytes) 000400 1787787686.361904 mapwrite 0x03260a thru 0x03ffff (0xd9f6 bytes) 000403 1787787686.454872 mapread 0x028264 thru 0x02fad2 (0x786f bytes) 000405 1787787686.464550 mapwrite 0x00c911 thru 0x01540e (0x8afe bytes) 000406 1787787686.534321 mapread 0x022d95 thru 0x02fa3e (0xccaa bytes) 000407 1787787686.535676 mapread 0x0105c1 thru 0x018bdf (0x861f bytes) 000410 1787787686.542865 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x54a2b299 (stale) Mirror 3: 0x54a2b299 (stale) Mirror 4: 0x62bba3c0 (stale) Mirror 5: 0x683a9c79 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2755ccce Mirror 6: 0x2755ccce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000411 1787787687.794507 write 0x0043e2 thru 0x0047b0 (0x3cf bytes) 000413 1787787687.913383 write 0x01907a thru 0x01fe7d (0x6e04 bytes) 000415 1787787687.928547 mapwrite 0x028bf0 thru 0x030ac7 (0x7ed8 bytes) 000416 1787787688.031391 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x54a2b299 (stale) Mirror 3: 0x54a2b299 (stale) Mirror 4: 0x62bba3c0 (stale) Mirror 5: 0x683a9c79 (stale) Mirror 6: 0x2755ccce (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7f2633c6 Mirror 7: 0x7f2633c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000417 1787787689.579993 mapwrite 0x002d4c thru 0x00e277 (0xb52c bytes) 000418 1787787689.810850 write 0x0263cd thru 0x03350e (0xd142 bytes) 000419 1787787689.838515 read 0x00293f thru 0x005e33 (0x34f5 bytes) 000422 1787787689.839562 mapwrite 0x03a765 thru 0x03ffff (0x589b bytes) 000423 1787787689.949107 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x54a2b299 (stale) Mirror 3: 0x54a2b299 (stale) Mirror 4: 0x62bba3c0 (stale) Mirror 5: 0x683a9c79 (stale) Mirror 6: 0x2755ccce (stale) Mirror 7: 0x7f2633c6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7ce6a583 Mirror 8: 0x7ce6a583 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000424 1787787691.557817 read 0x02850b thru 0x028c36 (0x72c bytes) 000425 1787787691.563573 write 0x0220c0 thru 0x02d0d7 (0xb018 bytes) 000426 1787787691.655323 trunc from 0x040000 to 0x03cab1 (0x354e bytes) 000427 1787787691.700409 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cab2): Mirror 2: 0x12163a15 (stale) Mirror 3: 0x12163a15 (stale) Mirror 5: 0xd4bc5e06 (stale) Mirror 6: 0x956c97a6 (stale) Mirror 7: 0x38e52eca (stale) Mirror 8: 0x2eb03c8d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cab2): Mirror 2: 0x12163a15 (stale) Mirror 3: 0x12163a15 (stale) Mirror 5: 0xd4bc5e06 (stale) Mirror 6: 0x956c97a6 (stale) Mirror 7: 0x38e52eca (stale) Mirror 8: 0x2eb03c8d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cab2: skipped 000428 1787787692.193605 mapread 0x030a85 thru 0x03aedf (0xa45b bytes) 000429 1787787692.220071 mapread 0x028b2a thru 0x035d55 (0xd22c bytes) 000431 1787787692.232820 mapwrite 0x01cf78 thru 0x021cb3 (0x4d3c bytes) 000432 1787787692.288785 read 0x00b161 thru 0x01a5cb (0xf46b bytes) 000433 1787787692.300429 mapwrite 0x01a1a2 thru 0x01afa4 (0xe03 bytes) 000435 1787787692.365621 mapwrite 0x007eec thru 0x00e47a (0x658f bytes) 000436 1787787692.431486 trunc from 0x03cab2 to 0x029e54 (0x12c5d bytes) 000438 1787787692.451798 trunc from 0x029e55 to 0x023694 (0x67c0 bytes) 000440 1787787692.478607 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x23695): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23695): Mirror 2: 0x5523917d (stale) Mirror 3: 0x5523917d (stale) Mirror 5: 0x3318b6f1 (stale) Mirror 6: 0x2a11939e (stale) Mirror 7: 0x8774e19 (stale) Mirror 8: 0xc0ea10e1 (stale) CRC-32 checksum value for chunk [0, 0x23695): Mirror 1: 0x33401406 Mirror 9: 0x33401406 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23695: skipped 000441 1787787693.352583 mapwrite 0x0154e6 thru 0x0166eb (0x1206 bytes) 000442 1787787693.528815 trunc from 0x023695 to 0x00222b (0x21469 bytes) 000444 1787787693.577505 trunc from 0x00222c to 0x036b24 (0x348f9 bytes) 000445 1787787693.590980 mapread 0x027e8b thru 0x02fbd3 (0x7d49 bytes) 000446 1787787693.602155 read 0x035e9c thru 0x036b24 (0xc89 bytes) 000447 1787787693.608214 mapread 0x02e9cb thru 0x030f44 (0x257a bytes) 000448 1787787693.616928 mapwrite 0x01622a thru 0x0203d0 (0xa1a7 bytes) 000449 1787787693.712825 trunc from 0x036b25 to 0x018637 (0x1e4ed bytes) 000450 1787787693.744989 mapwrite 0x00bca8 thru 0x01a4ff (0xe858 bytes) 000453 1787787693.839470 read 0x01746b thru 0x01a4ff (0x3095 bytes) 000454 1787787693.840013 read 0x012afe thru 0x01a4ff (0x7a02 bytes) 000457 1787787693.840930 write 0x0318fc thru 0x034e97 (0x359c bytes) 000459 1787787693.842810 write 0x01f5c4 thru 0x028455 (0x8e92 bytes) 000460 1787787693.856981 write 0x02a7d2 thru 0x02d218 (0x2a47 bytes) 000461 1787787693.858372 read 0x00dbaa thru 0x010c02 (0x3059 bytes) 000462 1787787693.859506 write 0x018a3c thru 0x022e56 (0xa41b bytes) 000464 1787787693.862319 read 0x00244c thru 0x00652b (0x40e0 bytes) 000466 1787787693.873173 write 0x02f664 thru 0x03c03b (0xc9d8 bytes) 000467 1787787693.874634 trunc from 0x03c03c to 0x024561 (0x17ada bytes) 000468 1787787693.903014 mapwrite 0x01fb30 thru 0x02e5c7 (0xea98 bytes) 000472 1787787694.006921 mapread 0x00cc25 thru 0x015301 (0x86dd bytes) 000477 1787787694.016361 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2e5c8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e5c8): Mirror 2: 0x6315eff3 (stale) Mirror 3: 0x6315eff3 (stale) Mirror 5: 0xec5982f0 (stale) Mirror 6: 0x5d6b395a (stale) Mirror 7: 0xfc15c001 (stale) Mirror 8: 0x4d0f82e1 (stale) Mirror 9: 0x5bc14026 (stale) CRC-32 checksum value for chunk [0, 0x2e5c8): Mirror 1: 0x40c6c826 Mirror 10: 0x40c6c826 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e5c8: skipped 000478 1787787695.250686 read 0x01fc14 thru 0x027058 (0x7445 bytes) 000479 1787787695.266459 mapwrite 0x00c32a thru 0x01530d (0x8fe4 bytes) 000480 1787787695.508658 write 0x011ff1 thru 0x01c398 (0xa3a8 bytes) 000481 1787787695.516536 mapread 0x01d971 thru 0x01e92c (0xfbc bytes) 000482 1787787695.529870 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e5c8): Mirror 2: 0x6315eff3 (stale) Mirror 3: 0x6315eff3 (stale) Mirror 5: 0xec5982f0 (stale) Mirror 6: 0x5d6b395a (stale) Mirror 8: 0x4d0f82e1 (stale) Mirror 9: 0x5bc14026 (stale) Mirror 10: 0x40c6c826 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e5c8): Mirror 2: 0x6315eff3 (stale) Mirror 3: 0x6315eff3 (stale) Mirror 5: 0xec5982f0 (stale) Mirror 6: 0x5d6b395a (stale) Mirror 8: 0x4d0f82e1 (stale) Mirror 9: 0x5bc14026 (stale) Mirror 10: 0x40c6c826 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e5c8: skipped 000486 1787787696.047845 mapread 0x02c11c thru 0x02e5c7 (0x24ac bytes) 000487 1787787696.072851 write 0x013d61 thru 0x01baac (0x7d4c bytes) 000488 1787787696.088938 mapread 0x018a3b thru 0x01fdbe (0x7384 bytes) 000489 1787787696.091804 trunc from 0x02e5c8 to 0x0379b5 (0x93ee bytes) 000493 1787787696.115715 write 0x025faf thru 0x031d26 (0xbd78 bytes) 000496 1787787696.129727 mapread 0x025b5b thru 0x027a43 (0x1ee9 bytes) 000497 1787787696.130399 mapread 0x01597c thru 0x02132f (0xb9b4 bytes) 000498 1787787696.139991 mapread 0x025fc2 thru 0x03241a (0xc459 bytes) 000501 1787787696.148707 mapread 0x00a8b3 thru 0x0152e5 (0xaa33 bytes) 000502 1787787696.161681 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x379b6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x379b6): Mirror 2: 0x939ed2b8 (stale) Mirror 3: 0x939ed2b8 (stale) Mirror 5: 0xbd60f077 (stale) Mirror 6: 0xec37fba8 (stale) Mirror 8: 0x9fe83c81 (stale) Mirror 9: 0x98195314 (stale) Mirror 10: 0xc574e6c5 (stale) CRC-32 checksum value for chunk [0, 0x379b6): Mirror 1: 0x195628a3 Mirror 11: 0x195628a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x379b6: skipped 000503 1787787697.443852 mapwrite 0x0372e4 thru 0x03a812 (0x352f bytes) 000506 1787787697.634438 write 0x01a443 thru 0x01b1fa (0xdb8 bytes) 000507 1787787697.644662 read 0x012b67 thru 0x02128b (0xe725 bytes) 000509 1787787697.659773 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3a813): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a813): Mirror 2: 0xc59761f0 (stale) Mirror 3: 0xc59761f0 (stale) Mirror 5: 0xe44dd278 (stale) Mirror 6: 0x64e644a1 (stale) Mirror 8: 0xb4e2d360 (stale) Mirror 9: 0xef73c6bf (stale) Mirror 10: 0xebf43515 (stale) Mirror 11: 0xfa909d43 (stale) CRC-32 checksum value for chunk [0, 0x3a813): Mirror 1: 0x31db9396 Mirror 12: 0x31db9396 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a813: skipped 000510 1787787698.988615 mapread 0x02a214 thru 0x039c7b (0xfa68 bytes) 000511 1787787699.015674 trunc from 0x03a813 to 0x036cc5 (0x3b4d bytes) 000512 1787787699.103544 trunc from 0x036cc6 to 0x023eff (0x12dc6 bytes) 000513 1787787699.127601 mapwrite 0x01a92f thru 0x01b194 (0x866 bytes) 000514 1787787699.208013 write 0x022170 thru 0x031639 (0xf4ca bytes) 000515 1787787699.209514 trunc from 0x03163a to 0x0184e5 (0x19154 bytes) 000516 1787787699.232029 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x184e6): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x184e6): Mirror 2: 0x1072858a (stale) Mirror 3: 0x1072858a (stale) Mirror 5: 0x1072858a (stale) Mirror 6: 0x84c539bc (stale) Mirror 8: 0xff6ce1c3 (stale) Mirror 9: 0x2845dd3a (stale) Mirror 10: 0x7d84ac0b (stale) Mirror 11: 0xfe934e24 (stale) Mirror 12: 0xfe934e24 (stale) CRC-32 checksum value for chunk [0, 0x184e6): Mirror 1: 0xfe934e24 Mirror 13: 0xfe934e24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x184e6: skipped 000517 1787787700.345863 write 0x038a27 thru 0x03ff0a (0x74e4 bytes) 000518 1787787700.430480 mapwrite 0x026034 thru 0x02eab5 (0x8a82 bytes) 000520 1787787700.495354 read 0x03aeb8 thru 0x03fdb8 (0x4f01 bytes) 000521 1787787700.496744 write 0x031d83 thru 0x034ad2 (0x2d50 bytes) 000526 1787787700.510716 mapread 0x02e596 thru 0x032a37 (0x44a2 bytes) 000527 1787787700.511811 trunc from 0x03ff0b to 0x006856 (0x396b4 bytes) 000528 1787787700.538627 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x6857): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6857): Mirror 2: 0x87421693 (stale) Mirror 3: 0x87421693 (stale) Mirror 5: 0x87421693 (stale) Mirror 6: 0x87421693 (stale) Mirror 8: 0x5b1d1933 (stale) Mirror 9: 0x5b1d1933 (stale) Mirror 10: 0x915bfbee (stale) Mirror 11: 0x915bfbee (stale) Mirror 12: 0x915bfbee (stale) Mirror 13: 0x915bfbee (stale) CRC-32 checksum value for chunk [0, 0x6857): Mirror 1: 0x915bfbee Mirror 14: 0x915bfbee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6857: skipped 000530 1787787701.585713 write 0x024f85 thru 0x026385 (0x1401 bytes) 000532 1787787701.665260 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x26386): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26386): Mirror 2: 0x54d6e03e (stale) Mirror 3: 0x54d6e03e (stale) Mirror 5: 0xf3934d41 (stale) Mirror 6: 0x7d77e07a (stale) Mirror 8: 0xfab3478c (stale) Mirror 9: 0x8e3e5647 (stale) Mirror 10: 0xe5ccb73a (stale) Mirror 11: 0x407d2a87 (stale) Mirror 12: 0x7167411b (stale) Mirror 13: 0xc58bec11 (stale) Mirror 14: 0x788d6480 (stale) CRC-32 checksum value for chunk [0, 0x26386): Mirror 1: 0x285725e4 Mirror 15: 0x285725e4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26386: skipped 000533 1787787702.975668 mapwrite 0x01015d thru 0x010f6e (0xe12 bytes) 000535 1787787703.186038 mapwrite 0x027ad5 thru 0x02b021 (0x354d bytes) 000536 1787787703.341310 mapread 0x012373 thru 0x0164a8 (0x4136 bytes) 000537 1787787703.361787 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b022): Mirror 2: 0xf3c5bd52 (stale) Mirror 3: 0xf3c5bd52 (stale) Mirror 5: 0x345d9d70 (stale) Mirror 6: 0x792fc43f (stale) Mirror 8: 0xcae61bb (stale) Mirror 9: 0x9f096a5 (stale) Mirror 10: 0xda3907a6 (stale) Mirror 12: 0x9969f5c (stale) Mirror 13: 0xf175b092 (stale) Mirror 14: 0x5fbe26f6 (stale) Mirror 15: 0x8a97b6b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b022): Mirror 2: 0xf3c5bd52 (stale) Mirror 3: 0xf3c5bd52 (stale) Mirror 5: 0x345d9d70 (stale) Mirror 6: 0x792fc43f (stale) Mirror 8: 0xcae61bb (stale) Mirror 9: 0x9f096a5 (stale) Mirror 10: 0xda3907a6 (stale) Mirror 12: 0x9969f5c (stale) Mirror 13: 0xf175b092 (stale) Mirror 14: 0x5fbe26f6 (stale) Mirror 15: 0x8a97b6b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b022: skipped 000538 1787787704.557231 write 0x02451e thru 0x031f4b (0xda2e bytes) 000540 1787787704.567949 read 0x004e71 thru 0x011280 (0xc410 bytes) 000541 1787787704.575675 mapwrite 0x0198ba thru 0x0200cf (0x6816 bytes) 000544 1787787704.630229 write 0x007cc9 thru 0x00b3dc (0x3714 bytes) 000545 1787787704.631342 read 0x01f63d thru 0x023db6 (0x477a bytes) 000547 1787787704.632981 write 0x03e2b4 thru 0x03f982 (0x16cf bytes) 000551 1787787704.635950 mapread 0x035edb thru 0x036e82 (0xfa8 bytes) 000552 1787787704.644950 mapwrite 0x0248f2 thru 0x0319f5 (0xd104 bytes) 000554 1787787704.698557 MIRROR_RESYNC 12 mirrors 000555 1787787710.173025 mapread 0x016eca thru 0x01f3da (0x8511 bytes) 000556 1787787710.217680 read 0x002e39 thru 0x0084d7 (0x569f bytes) 000557 1787787710.224802 mapread 0x03a54c thru 0x03f982 (0x5437 bytes) 000559 1787787710.234330 mapwrite 0x023e8c thru 0x02d008 (0x917d bytes) 000562 1787787710.509883 mapwrite 0x01e7a8 thru 0x027e6d (0x96c6 bytes) 000563 1787787710.576487 mapread 0x02e56f thru 0x03383f (0x52d1 bytes) 000564 1787787710.578001 trunc from 0x03f983 to 0x01ddde (0x21ba4 bytes) 000565 1787787710.635867 mapwrite 0x012621 thru 0x019a75 (0x7455 bytes) 000567 1787787710.710269 read 0x00bc37 thru 0x00d857 (0x1c21 bytes) 000568 1787787710.715263 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1dddf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dddf): Mirror 2: 0x61c9fa10 (stale) Mirror 3: 0x61c9fa10 (stale) Mirror 5: 0x61c9fa10 (stale) Mirror 6: 0x61c9fa10 (stale) Mirror 8: 0x61c9fa10 (stale) Mirror 9: 0x61c9fa10 (stale) Mirror 10: 0x61c9fa10 (stale) Mirror 12: 0x61c9fa10 (stale) Mirror 13: 0x61c9fa10 (stale) Mirror 14: 0x61c9fa10 (stale) Mirror 15: 0x61c9fa10 (stale) CRC-32 checksum value for chunk [0, 0x1dddf): Mirror 1: 0xc5a050ca Mirror 16: 0xc5a050ca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dddf: skipped 000569 1787787712.495017 mapwrite 0x021dae thru 0x028912 (0x6b65 bytes) 000570 1787787712.726611 mapread 0x00f9d9 thru 0x013898 (0x3ec0 bytes) 000571 1787787712.736965 mapread 0x010669 thru 0x01638f (0x5d27 bytes) 000573 1787787712.742089 read 0x02089b thru 0x028912 (0x8078 bytes) 000574 1787787712.747842 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28913): Mirror 2: 0x27a068d0 (stale) Mirror 3: 0x27a068d0 (stale) Mirror 5: 0x27a068d0 (stale) Mirror 6: 0x27a068d0 (stale) Mirror 8: 0x27a068d0 (stale) Mirror 9: 0x27a068d0 (stale) Mirror 10: 0x27a068d0 (stale) Mirror 12: 0x27a068d0 (stale) Mirror 13: 0x27a068d0 (stale) Mirror 14: 0x27a068d0 (stale) Mirror 16: 0xe18bdc2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28913): Mirror 2: 0x27a068d0 (stale) Mirror 3: 0x27a068d0 (stale) Mirror 5: 0x27a068d0 (stale) Mirror 6: 0x27a068d0 (stale) Mirror 8: 0x27a068d0 (stale) Mirror 9: 0x27a068d0 (stale) Mirror 10: 0x27a068d0 (stale) Mirror 12: 0x27a068d0 (stale) Mirror 13: 0x27a068d0 (stale) Mirror 14: 0x27a068d0 (stale) Mirror 16: 0xe18bdc2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28913: skipped 000576 1787787713.521379 write 0x00e254 thru 0x0169dc (0x8789 bytes) 000577 1787787713.536116 read 0x018086 thru 0x020be7 (0x8b62 bytes) 000578 1787787713.544005 mapwrite 0x03d0c8 thru 0x03ffff (0x2f38 bytes) 000579 1787787713.655998 mapwrite 0x023a5e thru 0x02cabb (0x905e bytes) 000580 1787787713.718850 mapread 0x01a7a1 thru 0x0263c1 (0xbc21 bytes) 000581 1787787713.719659 write 0x01f19a thru 0x020af5 (0x195c bytes) 000583 1787787713.720733 write 0x00e63c thru 0x010d45 (0x270a bytes) 000584 1787787713.721399 write 0x02fbc1 thru 0x038fec (0x942c bytes) 000586 1787787713.725552 MIRROR_RESYNC 12 mirrors 000587 1787787715.671539 trunc from 0x040000 to 0x00e18f (0x31e70 bytes) 000588 1787787715.804663 trunc from 0x00e190 to 0x0295c4 (0x1b435 bytes) 000593 1787787715.844327 write 0x032b76 thru 0x03e913 (0xbd9e bytes) 000594 1787787715.850344 read 0x002174 thru 0x00b7c1 (0x964e bytes) 000595 1787787715.862940 write 0x00e414 thru 0x00eea9 (0xa96 bytes) 000596 1787787715.872112 write 0x03896d thru 0x03e362 (0x59f6 bytes) 000597 1787787715.873257 write 0x002b66 thru 0x00c67e (0x9b19 bytes) 000599 1787787715.888035 write 0x02dce7 thru 0x033e34 (0x614e bytes) 000600 1787787715.896239 trunc from 0x03e914 to 0x00be0a (0x32b09 bytes) 000601 1787787715.927098 mapwrite 0x01e0f9 thru 0x02c122 (0xe02a bytes) 000603 1787787716.007598 read 0x008263 thru 0x00b962 (0x3700 bytes) 000607 1787787716.008938 mapwrite 0x036d39 thru 0x03ffff (0x92c7 bytes) 000608 1787787716.089241 read 0x038508 thru 0x03ac13 (0x270c bytes) 000609 1787787716.089824 write 0x019622 thru 0x022d8a (0x9769 bytes) 000611 1787787716.092529 mapwrite 0x016e11 thru 0x0182f6 (0x14e6 bytes) 000613 1787787716.194923 trunc from 0x040000 to 0x032c61 (0xd39e bytes) 000615 1787787716.219016 trunc from 0x032c62 to 0x038151 (0x54f0 bytes) 000618 1787787716.244885 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38152): Mirror 2: 0x744e4b61 (stale) Mirror 3: 0x744e4b61 (stale) Mirror 5: 0x744e4b61 (stale) Mirror 6: 0x744e4b61 (stale) Mirror 9: 0x744e4b61 (stale) Mirror 10: 0x744e4b61 (stale) Mirror 12: 0x744e4b61 (stale) Mirror 13: 0x744e4b61 (stale) Mirror 14: 0x744e4b61 (stale) Mirror 16: 0x744e4b61 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38152): Mirror 2: 0x744e4b61 (stale) Mirror 3: 0x744e4b61 (stale) Mirror 5: 0x744e4b61 (stale) Mirror 6: 0x744e4b61 (stale) Mirror 9: 0x744e4b61 (stale) Mirror 10: 0x744e4b61 (stale) Mirror 12: 0x744e4b61 (stale) Mirror 13: 0x744e4b61 (stale) Mirror 14: 0x744e4b61 (stale) Mirror 16: 0x744e4b61 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38152: skipped 000620 1787787716.981733 trunc from 0x038152 to 0x00617b (0x31fd6 bytes) 000623 1787787716.999620 read 0x000e67 thru 0x00617b (0x5315 bytes) 000625 1787787717.013027 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x617c): Mirror 2: 0xcd17a72f (stale) Mirror 3: 0xcd17a72f (stale) Mirror 5: 0xcd17a72f (stale) Mirror 6: 0xcd17a72f (stale) Mirror 9: 0xcd17a72f (stale) Mirror 12: 0xcd17a72f (stale) Mirror 13: 0xcd17a72f (stale) Mirror 14: 0xcd17a72f (stale) Mirror 16: 0xcd17a72f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x617c): Mirror 2: 0xcd17a72f (stale) Mirror 3: 0xcd17a72f (stale) Mirror 5: 0xcd17a72f (stale) Mirror 6: 0xcd17a72f (stale) Mirror 9: 0xcd17a72f (stale) Mirror 12: 0xcd17a72f (stale) Mirror 13: 0xcd17a72f (stale) Mirror 14: 0xcd17a72f (stale) Mirror 16: 0xcd17a72f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x617c: skipped 000627 1787787717.487725 mapread 0x003141 thru 0x00617b (0x303b bytes) 000628 1787787717.507888 write 0x03d7ca thru 0x03ffff (0x2836 bytes) 000629 1787787717.513793 MIRROR_RESYNC 10 mirrors 000631 1787787720.067204 write 0x00a158 thru 0x011639 (0x74e2 bytes) 000634 1787787720.161652 MIRROR_RESYNC 10 mirrors 000635 1787787722.732602 mapread 0x0094da thru 0x017328 (0xde4f bytes) 000636 1787787722.758790 trunc from 0x040000 to 0x006e40 (0x391bf bytes) 000637 1787787722.853803 mapread 0x0065d2 thru 0x006e40 (0x86f bytes) 000638 1787787722.861048 mapread 0x0069af thru 0x006b12 (0x164 bytes) 000644 1787787722.864345 trunc from 0x006e41 to 0x02e8ed (0x27aad bytes) 000645 1787787722.880507 read 0x00c96c thru 0x014a2a (0x80bf bytes) 000647 1787787722.889469 read 0x01ad62 thru 0x026105 (0xb3a4 bytes) 000648 1787787722.906157 mapwrite 0x021b1d thru 0x031935 (0xfe19 bytes) 000651 1787787722.989319 read 0x01a24a thru 0x01d109 (0x2ec0 bytes) 000652 1787787722.990615 write 0x00c149 thru 0x00cf95 (0xe4d bytes) 000653 1787787722.991816 trunc from 0x031936 to 0x02d126 (0x480f bytes) 000654 1787787723.013594 mapwrite 0x027e7e thru 0x036550 (0xe6d3 bytes) 000656 1787787723.102591 mapread 0x02199d thru 0x021de9 (0x44d bytes) 000657 1787787723.103460 trunc from 0x036551 to 0x025156 (0x113fa bytes) 000658 1787787723.117511 mapwrite 0x00d5e7 thru 0x01d3a9 (0xfdc3 bytes) 000660 1787787723.161804 trunc from 0x025157 to 0x02fd6b (0xac15 bytes) 000661 1787787723.192965 trunc from 0x02fd6c to 0x02edcf (0xf9c bytes) 000663 1787787723.220636 read 0x01d69a thru 0x02c32b (0xec92 bytes) 000667 1787787723.242627 read 0x0262b6 thru 0x02edcf (0x8b1a bytes) 000668 1787787723.243819 read 0x0204a6 thru 0x022be2 (0x273d bytes) 000672 1787787723.246779 mapread 0x02364e thru 0x02edcf (0xb782 bytes) 000673 1787787723.254455 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2edd0): Mirror 2: 0x48d62f4b (stale) Mirror 3: 0x48d62f4b (stale) Mirror 5: 0x48d62f4b (stale) Mirror 6: 0x48d62f4b (stale) Mirror 9: 0x48d62f4b (stale) Mirror 13: 0x48d62f4b (stale) Mirror 14: 0x48d62f4b (stale) Mirror 16: 0x48d62f4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edd0): Mirror 2: 0x48d62f4b (stale) Mirror 3: 0x48d62f4b (stale) Mirror 5: 0x48d62f4b (stale) Mirror 6: 0x48d62f4b (stale) Mirror 9: 0x48d62f4b (stale) Mirror 13: 0x48d62f4b (stale) Mirror 14: 0x48d62f4b (stale) Mirror 16: 0x48d62f4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edd0: skipped 000674 1787787723.946615 mapread 0x01ce99 thru 0x028a22 (0xbb8a bytes) 000675 1787787723.981608 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2edd0): Mirror 2: 0x48d62f4b (stale) Mirror 3: 0x48d62f4b (stale) Mirror 5: 0x48d62f4b (stale) Mirror 6: 0x48d62f4b (stale) Mirror 13: 0x48d62f4b (stale) Mirror 14: 0x48d62f4b (stale) Mirror 16: 0x48d62f4b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2edd0): Mirror 2: 0x48d62f4b (stale) Mirror 3: 0x48d62f4b (stale) Mirror 5: 0x48d62f4b (stale) Mirror 6: 0x48d62f4b (stale) Mirror 13: 0x48d62f4b (stale) Mirror 14: 0x48d62f4b (stale) Mirror 16: 0x48d62f4b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2edd0: skipped 000676 1787787724.502881 read 0x005ec8 thru 0x011f79 (0xc0b2 bytes) 000677 1787787724.511052 trunc from 0x02edd0 to 0x000fc5 (0x2de0a bytes) 000681 1787787724.540976 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfc6): Mirror 2: 0x93911714 (stale) Mirror 3: 0x93911714 (stale) Mirror 5: 0x93911714 (stale) Mirror 6: 0x93911714 (stale) Mirror 13: 0x93911714 (stale) Mirror 16: 0x93911714 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfc6): Mirror 2: 0x93911714 (stale) Mirror 3: 0x93911714 (stale) Mirror 5: 0x93911714 (stale) Mirror 6: 0x93911714 (stale) Mirror 13: 0x93911714 (stale) Mirror 16: 0x93911714 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfc6: skipped 000682 1787787724.956485 mapwrite 0x026f43 thru 0x030b6c (0x9c2a bytes) 000685 1787787725.069085 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x30b6d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b6d): Mirror 2: 0x3f45fc60 (stale) Mirror 3: 0x3f45fc60 (stale) Mirror 5: 0x3f45fc60 (stale) Mirror 6: 0x3f45fc60 (stale) Mirror 13: 0x3f45fc60 (stale) Mirror 16: 0x3f45fc60 (stale) CRC-32 checksum value for chunk [0, 0x30b6d): Mirror 1: 0xecfd2106 Mirror 17: 0xecfd2106 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b6d: skipped 000686 1787787726.049257 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x30b6d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b6d): Mirror 2: 0x3f45fc60 (stale) Mirror 3: 0x3f45fc60 (stale) Mirror 5: 0x3f45fc60 (stale) Mirror 6: 0x3f45fc60 (stale) Mirror 13: 0x3f45fc60 (stale) Mirror 16: 0x3f45fc60 (stale) CRC-32 checksum value for chunk [0, 0x30b6d): Mirror 1: 0xecfd2106 Mirror 17: 0xecfd2106 Mirror 18: 0xecfd2106 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b6d: skipped 000688 1787787727.501899 trunc from 0x030b6d to 0x02d8af (0x32bd bytes) 000691 1787787727.623501 read 0x0024ed thru 0x00c76f (0xa283 bytes) 000693 1787787727.659681 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2d8b0): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d8b0): Mirror 2: 0xa892496e (stale) Mirror 3: 0xa892496e (stale) Mirror 5: 0xa892496e (stale) Mirror 6: 0xa892496e (stale) Mirror 13: 0xa892496e (stale) Mirror 16: 0xa892496e (stale) Mirror 17: 0xb205987e (stale) Mirror 18: 0xb205987e (stale) CRC-32 checksum value for chunk [0, 0x2d8b0): Mirror 1: 0xb205987e Mirror 19: 0xb205987e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d8b0: skipped 000694 1787787729.595994 mapread 0x006c67 thru 0x01358e (0xc928 bytes) 000697 1787787729.627184 trunc from 0x02d8b0 to 0x0156c1 (0x181ee bytes) 000698 1787787729.760106 trunc from 0x0156c2 to 0x0375a6 (0x21ee5 bytes) 000699 1787787729.797493 mapwrite 0x0050c0 thru 0x00c460 (0x73a1 bytes) 000700 1787787729.907098 mapread 0x010307 thru 0x0186a9 (0x83a3 bytes) 000702 1787787729.916413 mapwrite 0x0388e0 thru 0x03ffff (0x7720 bytes) 000704 1787787730.113405 read 0x039fb7 thru 0x03ffff (0x6049 bytes) 000705 1787787730.116278 write 0x0333d3 thru 0x034619 (0x1247 bytes) 000707 1787787730.119459 trunc from 0x040000 to 0x027d82 (0x1827d bytes) 000708 1787787730.153426 mapread 0x01da80 thru 0x020990 (0x2f11 bytes) 000709 1787787730.169568 write 0x01134a thru 0x0179fb (0x66b2 bytes) 000710 1787787730.172467 mapwrite 0x01b7b7 thru 0x023a9b (0x82e5 bytes) 000711 1787787730.333345 mapread 0x0101ab thru 0x0105b7 (0x40d bytes) 000713 1787787730.334628 mapwrite 0x006c2a thru 0x0098b2 (0x2c89 bytes) 000715 1787787730.501464 read 0x011379 thru 0x01f916 (0xe59e bytes) 000716 1787787730.502320 read 0x022347 thru 0x027d82 (0x5a3c bytes) 000717 1787787730.537573 write 0x00619e thru 0x00e2c0 (0x8123 bytes) 000718 1787787730.546645 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27d83): Mirror 2: 0xd2dad468 (stale) Mirror 3: 0xd2dad468 (stale) Mirror 5: 0xd2dad468 (stale) Mirror 6: 0xd2dad468 (stale) Mirror 13: 0xd2dad468 (stale) Mirror 16: 0xd2dad468 (stale) Mirror 18: 0x67b271ed (stale) Mirror 19: 0x67b271ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27d83): Mirror 2: 0xd2dad468 (stale) Mirror 3: 0xd2dad468 (stale) Mirror 5: 0xd2dad468 (stale) Mirror 6: 0xd2dad468 (stale) Mirror 13: 0xd2dad468 (stale) Mirror 16: 0xd2dad468 (stale) Mirror 18: 0x67b271ed (stale) Mirror 19: 0x67b271ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27d83: skipped 000719 1787787731.493751 mapread 0x016a4a thru 0x024a06 (0xdfbd bytes) 000721 1787787731.574569 trunc from 0x027d83 to 0x03426c (0xc4ea bytes) 000722 1787787731.609412 trunc from 0x03426d to 0x004f8f (0x2f2dd bytes) 000724 1787787731.688066 MIRROR_RESYNC 9 mirrors 000728 1787787734.512285 write 0x00d7b8 thru 0x01b5e5 (0xde2e bytes) 000729 1787787734.755716 trunc from 0x01b5e6 to 0x025ebb (0xa8d6 bytes) 000730 1787787734.812722 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25ebc): Mirror 2: 0xa4ac7490 (stale) Mirror 3: 0xa4ac7490 (stale) Mirror 5: 0xa4ac7490 (stale) Mirror 13: 0xa4ac7490 (stale) Mirror 16: 0xa4ac7490 (stale) Mirror 18: 0xa4ac7490 (stale) Mirror 19: 0xa4ac7490 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25ebc): Mirror 2: 0xa4ac7490 (stale) Mirror 3: 0xa4ac7490 (stale) Mirror 5: 0xa4ac7490 (stale) Mirror 13: 0xa4ac7490 (stale) Mirror 16: 0xa4ac7490 (stale) Mirror 18: 0xa4ac7490 (stale) Mirror 19: 0xa4ac7490 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25ebc: skipped 000731 1787787735.786442 trunc from 0x025ebc to 0x008427 (0x1da94 bytes) 000732 1787787735.811257 mapwrite 0x03fffe thru 0x03ffff (0x2 bytes) 000733 1787787735.956460 write 0x01558c thru 0x01dbb7 (0x862c bytes) 000734 1787787735.972776 mapwrite 0x00dc30 thru 0x0187ee (0xabbf bytes) 000739 1787787736.066750 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcd6e7392 (stale) Mirror 3: 0xcd6e7392 (stale) Mirror 5: 0xcd6e7392 (stale) Mirror 13: 0xcd6e7392 (stale) Mirror 16: 0xcd6e7392 (stale) Mirror 18: 0xcd6e7392 (stale) Mirror 19: 0xcd6e7392 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc4a25cbe Mirror 20: 0xc4a25cbe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000740 1787787737.798964 mapread 0x013972 thru 0x01f58a (0xbc19 bytes) truncating to largest ever: 0x3f1e4 000742 1787787737.860607 trunc from 0x040000 to 0x03f1e3 (0xe1c bytes) 000743 1787787738.047981 write 0x0389b1 thru 0x03b393 (0x29e3 bytes) 000745 1787787738.066330 mapwrite 0x00701d thru 0x00ccef (0x5cd3 bytes) 000746 1787787738.192221 read 0x025c62 thru 0x032324 (0xc6c3 bytes) 000747 1787787738.210300 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f1e4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1e4): Mirror 2: 0x42a15b2e (stale) Mirror 3: 0x42a15b2e (stale) Mirror 5: 0x42a15b2e (stale) Mirror 13: 0x42a15b2e (stale) Mirror 16: 0x42a15b2e (stale) Mirror 18: 0x42a15b2e (stale) Mirror 19: 0x42a15b2e (stale) Mirror 20: 0x8cf18eee (stale) CRC-32 checksum value for chunk [0, 0x3f1e4): Mirror 1: 0x1d0f55a0 Mirror 21: 0x1d0f55a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1e4: skipped 000748 1787787739.476026 mapwrite 0x00d86a thru 0x00e498 (0xc2f bytes) 000749 1787787739.633102 write 0x027cb0 thru 0x02db64 (0x5eb5 bytes) 000750 1787787739.647204 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3f1e4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1e4): Mirror 2: 0x42a15b2e (stale) Mirror 3: 0x42a15b2e (stale) Mirror 5: 0x42a15b2e (stale) Mirror 13: 0x42a15b2e (stale) Mirror 16: 0x42a15b2e (stale) Mirror 18: 0x42a15b2e (stale) Mirror 19: 0x42a15b2e (stale) Mirror 20: 0x8cf18eee (stale) Mirror 21: 0x1d0f55a0 (stale) CRC-32 checksum value for chunk [0, 0x3f1e4): Mirror 1: 0x868ba39c Mirror 22: 0x868ba39c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1e4: skipped 000752 1787787740.997694 read 0x035c50 thru 0x03f1e3 (0x9594 bytes) 000753 1787787741.008087 trunc from 0x03f1e4 to 0x01a67e (0x24b65 bytes) 000754 1787787741.134771 read 0x00a75f thru 0x00ca49 (0x22eb bytes) 000755 1787787741.147975 mapread 0x017549 thru 0x018625 (0x10dd bytes) 000756 1787787741.150205 trunc from 0x01a67f to 0x020a63 (0x63e5 bytes) 000757 1787787741.197960 mapwrite 0x028cf2 thru 0x035d16 (0xd025 bytes) 000758 1787787741.312102 write 0x00bdb8 thru 0x00cc2f (0xe78 bytes) 000759 1787787741.313100 mapwrite 0x038ac1 thru 0x03e21f (0x575f bytes) 000762 1787787741.436380 trunc from 0x03e220 to 0x013c8f (0x2a590 bytes) 000763 1787787741.453032 read 0x00a4ec thru 0x013c8f (0x97a4 bytes) 000766 1787787741.454658 write 0x0271be thru 0x031c0e (0xaa51 bytes) 000768 1787787741.460817 mapread 0x012091 thru 0x014bdb (0x2b4b bytes) 000769 1787787741.470968 trunc from 0x031c0f to 0x03eb92 (0xcf84 bytes) 000770 1787787741.490175 mapread 0x00385b thru 0x00fdbc (0xc562 bytes) 000771 1787787741.491186 mapwrite 0x03b297 thru 0x03ffff (0x4d69 bytes) 000772 1787787741.575921 read 0x026f67 thru 0x034bd2 (0xdc6c bytes) 000773 1787787741.577292 write 0x03b840 thru 0x03dd75 (0x2536 bytes) 000774 1787787741.578809 read 0x00735a thru 0x014416 (0xd0bd bytes) 000775 1787787741.582318 read 0x02ea11 thru 0x02fbc3 (0x11b3 bytes) 000777 1787787741.583596 mapwrite 0x01568c thru 0x01ebd6 (0x954b bytes) 000778 1787787741.640762 trunc from 0x040000 to 0x0116cd (0x2e932 bytes) 000779 1787787741.664684 write 0x02354c thru 0x02f045 (0xbafa bytes) 000780 1787787741.666116 mapwrite 0x03a65b thru 0x03ffff (0x59a5 bytes) 000781 1787787741.757612 trunc from 0x040000 to 0x032f3b (0xd0c4 bytes) 000784 1787787741.789532 trunc from 0x032f3c to 0x00c670 (0x268cb bytes) 000785 1787787741.808125 mapwrite 0x037450 thru 0x038c47 (0x17f8 bytes) 000786 1787787741.920031 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x38c48): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c48): Mirror 2: 0xf7ca616e (stale) Mirror 3: 0xf7ca616e (stale) Mirror 5: 0xf7ca616e (stale) Mirror 13: 0xf7ca616e (stale) Mirror 16: 0xf7ca616e (stale) Mirror 18: 0xf7ca616e (stale) Mirror 19: 0xf7ca616e (stale) Mirror 20: 0x613774cb (stale) Mirror 21: 0x3cb2a522 (stale) Mirror 22: 0x20710da3 (stale) CRC-32 checksum value for chunk [0, 0x38c48): Mirror 1: 0x7dc746c7 Mirror 23: 0x7dc746c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c48: skipped 000787 1787787743.285130 mapread 0x000bdc thru 0x001a75 (0xe9a bytes) 000788 1787787743.313858 write 0x028077 thru 0x0315d1 (0x955b bytes) 000789 1787787743.506995 read 0x026c78 thru 0x02837f (0x1708 bytes) 000790 1787787743.518336 write 0x01f4d4 thru 0x02daef (0xe61c bytes) 000791 1787787743.544697 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x38c48): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c48): Mirror 2: 0xf7ca616e (stale) Mirror 3: 0xf7ca616e (stale) Mirror 5: 0xf7ca616e (stale) Mirror 13: 0xf7ca616e (stale) Mirror 16: 0xf7ca616e (stale) Mirror 18: 0xf7ca616e (stale) Mirror 19: 0xf7ca616e (stale) Mirror 20: 0x613774cb (stale) Mirror 21: 0x3cb2a522 (stale) Mirror 22: 0x20710da3 (stale) Mirror 23: 0x7dc746c7 (stale) CRC-32 checksum value for chunk [0, 0x38c48): Mirror 1: 0x15493ffd Mirror 24: 0x15493ffd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c48: skipped 000793 1787787745.082896 trunc from 0x038c48 to 0x011757 (0x274f0 bytes) 000797 1787787745.201124 mapread 0x006ee5 thru 0x00b55c (0x4678 bytes) 000800 1787787745.222373 mapread 0x0016fc thru 0x004bac (0x34b1 bytes) 000801 1787787745.223281 read 0x005c04 thru 0x00e5ed (0x89ea bytes) 000802 1787787745.224715 read 0x003f62 thru 0x011757 (0xd7f6 bytes) 000803 1787787745.238048 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x11758): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11758): Mirror 2: 0x18a84034 (stale) Mirror 3: 0x18a84034 (stale) Mirror 5: 0x18a84034 (stale) Mirror 13: 0x18a84034 (stale) Mirror 16: 0x18a84034 (stale) Mirror 18: 0x18a84034 (stale) Mirror 19: 0x18a84034 (stale) Mirror 20: 0x84e5f77f (stale) Mirror 21: 0x52888f3c (stale) Mirror 22: 0x18c21fa5 (stale) Mirror 23: 0x42a8cf50 (stale) Mirror 24: 0x42a8cf50 (stale) CRC-32 checksum value for chunk [0, 0x11758): Mirror 1: 0x42a8cf50 Mirror 25: 0x42a8cf50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11758: skipped 000805 1787787746.626479 mapwrite 0x02bf32 thru 0x02f419 (0x34e8 bytes) 000806 1787787746.921436 mapread 0x01bacf thru 0x01fed2 (0x4404 bytes) 000807 1787787746.932108 read 0x004b93 thru 0x01283e (0xdcac bytes) 000808 1787787746.940007 write 0x009138 thru 0x00b566 (0x242f bytes) 000809 1787787746.941306 trunc from 0x02f41a to 0x015347 (0x1a0d2 bytes) 000810 1787787746.978693 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15348): Mirror 2: 0xa853e2c0 (stale) Mirror 3: 0xa853e2c0 (stale) Mirror 5: 0xa853e2c0 (stale) Mirror 13: 0xa853e2c0 (stale) Mirror 16: 0xa853e2c0 (stale) Mirror 18: 0xa853e2c0 (stale) Mirror 19: 0xa853e2c0 (stale) Mirror 20: 0x959701b2 (stale) Mirror 21: 0xdb51e53 (stale) Mirror 23: 0xd38bff50 (stale) Mirror 24: 0xd38bff50 (stale) Mirror 25: 0xd38bff50 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15348): Mirror 2: 0xa853e2c0 (stale) Mirror 3: 0xa853e2c0 (stale) Mirror 5: 0xa853e2c0 (stale) Mirror 13: 0xa853e2c0 (stale) Mirror 16: 0xa853e2c0 (stale) Mirror 18: 0xa853e2c0 (stale) Mirror 19: 0xa853e2c0 (stale) Mirror 20: 0x959701b2 (stale) Mirror 21: 0xdb51e53 (stale) Mirror 23: 0xd38bff50 (stale) Mirror 24: 0xd38bff50 (stale) Mirror 25: 0xd38bff50 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15348: skipped 000811 1787787747.663019 write 0x0129bf thru 0x01c2d5 (0x9917 bytes) 000812 1787787747.669882 mapwrite 0x03d3e5 thru 0x03f59b (0x21b7 bytes) 000813 1787787747.740372 mapwrite 0x00f4ed thru 0x012b04 (0x3618 bytes) 000815 1787787747.777129 write 0x022058 thru 0x02b068 (0x9011 bytes) 000817 1787787747.786846 read 0x001eb1 thru 0x009043 (0x7193 bytes) 000818 1787787747.796730 read 0x022d27 thru 0x030b5a (0xde34 bytes) 000821 1787787747.813086 trunc from 0x03f59c to 0x012668 (0x2cf33 bytes) 000822 1787787747.838680 write 0x01e5ca thru 0x023b80 (0x55b7 bytes) 000823 1787787747.840230 mapwrite 0x03b973 thru 0x03ffff (0x468d bytes) 000824 1787787747.960354 trunc from 0x040000 to 0x021497 (0x1eb68 bytes) 000825 1787787747.981588 trunc from 0x021498 to 0x02521c (0x3d85 bytes) 000827 1787787748.005468 read 0x012ccb thru 0x0189a5 (0x5cdb bytes) 000831 1787787748.017932 trunc from 0x02521d to 0x0184ff (0xcd1d bytes) 000832 1787787748.033820 trunc from 0x018500 to 0x03ca77 (0x24578 bytes) 000833 1787787748.050217 trunc from 0x03ca78 to 0x0396a3 (0x33d4 bytes) 000836 1787787748.070104 mapwrite 0x037307 thru 0x0374a4 (0x19e bytes) 000838 1787787748.122219 trunc from 0x0396a4 to 0x03b3b3 (0x1d10 bytes) 000839 1787787748.141355 read 0x004447 thru 0x00e50c (0xa0c6 bytes) 000840 1787787748.142984 trunc from 0x03b3b4 to 0x008493 (0x32f20 bytes) 000841 1787787748.170063 mapwrite 0x01c111 thru 0x02b0be (0xefae bytes) 000842 1787787748.249485 trunc from 0x02b0bf to 0x01d46a (0xdc54 bytes) 000843 1787787748.275758 read 0x00e94d thru 0x013a37 (0x50eb bytes) 000844 1787787748.276396 mapread 0x01c51b thru 0x01d46a (0xf50 bytes) 000845 1787787748.277246 mapwrite 0x009858 thru 0x0165d5 (0xcd7e bytes) 000846 1787787748.328422 write 0x01b3f4 thru 0x01c0d0 (0xcdd bytes) 000847 1787787748.329989 write 0x023a97 thru 0x02ca17 (0x8f81 bytes) 000849 1787787748.332977 mapwrite 0x00ca4a thru 0x011eaf (0x5466 bytes) 000850 1787787748.386012 mapread 0x028c0b thru 0x02a342 (0x1738 bytes) 000853 1787787748.389310 mapread 0x02b662 thru 0x02ca17 (0x13b6 bytes) 000856 1787787748.391564 write 0x0148aa thru 0x017184 (0x28db bytes) 000861 1787787748.395833 read 0x015a8e thru 0x01bca4 (0x6217 bytes) 000864 1787787748.397702 read 0x029d0f thru 0x02a2aa (0x59c bytes) 000865 1787787748.398307 trunc from 0x02ca18 to 0x019bd2 (0x12e45 bytes) 000866 1787787748.424675 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19bd3): Mirror 2: 0xff506048 (stale) Mirror 3: 0xff506048 (stale) Mirror 5: 0xff506048 (stale) Mirror 13: 0xff506048 (stale) Mirror 16: 0xff506048 (stale) Mirror 18: 0xff506048 (stale) Mirror 19: 0xff506048 (stale) Mirror 20: 0x5f0d621f (stale) Mirror 21: 0xdc55bb97 (stale) Mirror 23: 0x8ac7a24 (stale) Mirror 24: 0x8ac7a24 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19bd3): Mirror 2: 0xff506048 (stale) Mirror 3: 0xff506048 (stale) Mirror 5: 0xff506048 (stale) Mirror 13: 0xff506048 (stale) Mirror 16: 0xff506048 (stale) Mirror 18: 0xff506048 (stale) Mirror 19: 0xff506048 (stale) Mirror 20: 0x5f0d621f (stale) Mirror 21: 0xdc55bb97 (stale) Mirror 23: 0x8ac7a24 (stale) Mirror 24: 0x8ac7a24 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19bd3: skipped 000867 1787787748.988195 read 0x00a708 thru 0x012df8 (0x86f1 bytes) 000870 1787787749.014521 read 0x00650d thru 0x00b2ea (0x4dde bytes) 000871 1787787749.030819 mapread 0x014475 thru 0x0166fb (0x2287 bytes) 000875 1787787749.035831 write 0x02f8e4 thru 0x034cb8 (0x53d5 bytes) 000876 1787787749.038818 trunc from 0x034cb9 to 0x02be3c (0x8e7c bytes) 000878 1787787749.069620 read 0x022745 thru 0x026a29 (0x42e5 bytes) 000880 1787787749.083536 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2be3d): Mirror 2: 0xae4d2fb3 (stale) Mirror 3: 0xae4d2fb3 (stale) Mirror 5: 0xae4d2fb3 (stale) Mirror 13: 0xae4d2fb3 (stale) Mirror 16: 0xae4d2fb3 (stale) Mirror 18: 0xae4d2fb3 (stale) Mirror 19: 0xae4d2fb3 (stale) Mirror 20: 0x8b54fb3e (stale) Mirror 21: 0x1b40e304 (stale) Mirror 24: 0xab44548c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2be3d): Mirror 2: 0xae4d2fb3 (stale) Mirror 3: 0xae4d2fb3 (stale) Mirror 5: 0xae4d2fb3 (stale) Mirror 13: 0xae4d2fb3 (stale) Mirror 16: 0xae4d2fb3 (stale) Mirror 18: 0xae4d2fb3 (stale) Mirror 19: 0xae4d2fb3 (stale) Mirror 20: 0x8b54fb3e (stale) Mirror 21: 0x1b40e304 (stale) Mirror 24: 0xab44548c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2be3d: skipped 000882 1787787749.595057 read 0x012ac3 thru 0x022949 (0xfe87 bytes) 000884 1787787749.607733 trunc from 0x02be3d to 0x010337 (0x1bb05 bytes) 000885 1787787749.625526 mapread 0x00336b thru 0x0035c6 (0x25c bytes) 000886 1787787749.631959 read 0x00fb5f thru 0x010337 (0x7d9 bytes) 000887 1787787749.641293 mapwrite 0x00107a thru 0x00d7cb (0xc752 bytes) 000889 1787787749.709986 mapread 0x002f30 thru 0x010337 (0xd408 bytes) 000891 1787787749.712291 mapwrite 0x00fbd6 thru 0x01d121 (0xd54c bytes) 000892 1787787749.790064 mapread 0x0086dc thru 0x0179a5 (0xf2ca bytes) 000893 1787787749.791404 read 0x008e2f thru 0x014bbf (0xbd91 bytes) 000895 1787787749.794056 read 0x017cbd thru 0x01d121 (0x5465 bytes) 000897 1787787749.795999 mapread 0x019d81 thru 0x01d121 (0x33a1 bytes) 000898 1787787749.802376 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d122): Mirror 2: 0x2e4ec5f1 (stale) Mirror 3: 0x2e4ec5f1 (stale) Mirror 5: 0x2e4ec5f1 (stale) Mirror 16: 0x2e4ec5f1 (stale) Mirror 18: 0x2e4ec5f1 (stale) Mirror 19: 0x2e4ec5f1 (stale) Mirror 20: 0x54750d42 (stale) Mirror 21: 0x9d037eb7 (stale) Mirror 24: 0x8bad7e6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d122): Mirror 2: 0x2e4ec5f1 (stale) Mirror 3: 0x2e4ec5f1 (stale) Mirror 5: 0x2e4ec5f1 (stale) Mirror 16: 0x2e4ec5f1 (stale) Mirror 18: 0x2e4ec5f1 (stale) Mirror 19: 0x2e4ec5f1 (stale) Mirror 20: 0x54750d42 (stale) Mirror 21: 0x9d037eb7 (stale) Mirror 24: 0x8bad7e6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d122: skipped 000900 1787787750.304820 MIRROR_RESYNC 10 mirrors 000903 1787787752.378879 MIRROR_RESYNC 10 mirrors 000904 1787787753.072689 mapread 0x00a72b thru 0x016404 (0xbcda bytes) 000905 1787787753.126194 mapread 0x00aeaf thru 0x01229e (0x73f0 bytes) 000906 1787787753.126696 mapread 0x0022e8 thru 0x00b633 (0x934c bytes) 000907 1787787753.144267 mapwrite 0x000e9a thru 0x008be1 (0x7d48 bytes) 000909 1787787753.437649 read 0x005b9e thru 0x007667 (0x1aca bytes) 000911 1787787753.439838 trunc from 0x01d122 to 0x00db44 (0xf5dd bytes) 000914 1787787753.467464 read 0x003898 thru 0x00db44 (0xa2ad bytes) 000915 1787787753.468019 mapread 0x00c49d thru 0x00db44 (0x16a8 bytes) 000917 1787787753.477240 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xdb45): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdb45): Mirror 2: 0xa44311f (stale) Mirror 3: 0xa44311f (stale) Mirror 5: 0xa44311f (stale) Mirror 16: 0xa44311f (stale) Mirror 18: 0xa44311f (stale) Mirror 19: 0xa44311f (stale) Mirror 20: 0xa44311f (stale) Mirror 21: 0xa44311f (stale) Mirror 24: 0xa44311f (stale) CRC-32 checksum value for chunk [0, 0xdb45): Mirror 1: 0x77468d6c Mirror 25: 0x77468d6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdb45: skipped 000918 1787787754.377371 read 0x005484 thru 0x00db44 (0x86c1 bytes) 000921 1787787754.404986 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0xdb45): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdb45): Mirror 2: 0xa44311f (stale) Mirror 3: 0xa44311f (stale) Mirror 5: 0xa44311f (stale) Mirror 16: 0xa44311f (stale) Mirror 18: 0xa44311f (stale) Mirror 19: 0xa44311f (stale) Mirror 20: 0xa44311f (stale) Mirror 21: 0xa44311f (stale) Mirror 24: 0xa44311f (stale) CRC-32 checksum value for chunk [0, 0xdb45): Mirror 1: 0x77468d6c Mirror 25: 0x77468d6c Mirror 26: 0x77468d6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdb45: skipped 000922 1787787755.474785 read 0x00cb69 thru 0x00db44 (0xfdc bytes) 000923 1787787755.480185 mapwrite 0x01e9d4 thru 0x02c56d (0xdb9a bytes) 000924 1787787755.711874 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2c56e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c56e): Mirror 2: 0x8c3a5195 (stale) Mirror 3: 0x8c3a5195 (stale) Mirror 5: 0x8c3a5195 (stale) Mirror 16: 0x8c3a5195 (stale) Mirror 18: 0x8c3a5195 (stale) Mirror 19: 0x8c3a5195 (stale) Mirror 20: 0x8c3a5195 (stale) Mirror 21: 0x8c3a5195 (stale) Mirror 24: 0x8c3a5195 (stale) Mirror 25: 0x95b9f748 (stale) Mirror 26: 0x95b9f748 (stale) CRC-32 checksum value for chunk [0, 0x2c56e): Mirror 1: 0x5c9fcf5c Mirror 27: 0x5c9fcf5c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c56e: skipped 000928 1787787757.141319 MIRROR_RESYNC 13 mirrors 000933 1787787760.979117 mapread 0x01c057 thru 0x02062e (0x45d8 bytes) 000934 1787787760.992922 mapread 0x026d3e thru 0x02c56d (0x5830 bytes) 000935 1787787761.001859 mapread 0x008e7b thru 0x016a7b (0xdc01 bytes) 000938 1787787761.018739 trunc from 0x02c56e to 0x00f0cd (0x1d4a0 bytes) 000940 1787787761.156604 write 0x00b3b2 thru 0x010bf2 (0x5841 bytes) 000943 1787787761.167053 mapwrite 0x0221d7 thru 0x026f44 (0x4d6e bytes) 000944 1787787761.353326 trunc from 0x026f45 to 0x02f75c (0x8818 bytes) 000946 1787787761.409555 MIRROR_RESYNC 13 mirrors 000949 1787787765.248954 mapwrite 0x007b89 thru 0x00f212 (0x768a bytes) 000950 1787787765.460919 trunc from 0x02f75d to 0x00274b (0x2d011 bytes) 000951 1787787765.493628 read 0x001b25 thru 0x00274b (0xc27 bytes) 000952 1787787765.494301 trunc from 0x00274c to 0x017871 (0x15126 bytes) 000954 1787787765.517924 write 0x02e73c thru 0x03aa0c (0xc2d1 bytes) 000955 1787787765.522761 trunc from 0x03aa0d to 0x00201c (0x389f0 bytes) 000956 1787787765.550600 mapread 0x0009a9 thru 0x00201c (0x1674 bytes) 000958 1787787765.567920 MIRROR_RESYNC 13 mirrors 000961 1787787767.637630 mapread 0x00066e thru 0x00201c (0x19af bytes) 000965 1787787767.674854 trunc from 0x00201d to 0x0394ad (0x37491 bytes) 000966 1787787767.812975 mapread 0x01d5ad thru 0x0285d7 (0xb02b bytes) 000967 1787787767.840816 write 0x025725 thru 0x031e85 (0xc761 bytes) 000969 1787787767.859917 read 0x01cde6 thru 0x027963 (0xab7e bytes) 000970 1787787767.862089 mapread 0x012382 thru 0x0206d0 (0xe34f bytes) 000972 1787787767.895308 mapread 0x01767b thru 0x01f379 (0x7cff bytes) 000973 1787787767.898804 read 0x02d6a8 thru 0x02fda2 (0x26fb bytes) 000975 1787787767.903914 mapread 0x01605e thru 0x02050b (0xa4ae bytes) 000976 1787787767.906752 mapread 0x02d0d8 thru 0x02f91f (0x2848 bytes) 000977 1787787767.908306 write 0x03d52c thru 0x03ffff (0x2ad4 bytes) 000981 1787787767.918552 write 0x004f02 thru 0x007a3a (0x2b39 bytes) 000982 1787787767.936774 mapread 0x01e6ab thru 0x01fce0 (0x1636 bytes) 000983 1787787767.938041 mapwrite 0x03941c thru 0x03ffff (0x6be4 bytes) 000984 1787787768.019786 trunc from 0x040000 to 0x029f5d (0x160a2 bytes) 000986 1787787768.052593 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29f5e): Mirror 2: 0x28749b2b (stale) Mirror 3: 0x28749b2b (stale) Mirror 5: 0x28749b2b (stale) Mirror 16: 0x28749b2b (stale) Mirror 18: 0x28749b2b (stale) Mirror 19: 0x28749b2b (stale) Mirror 20: 0x28749b2b (stale) Mirror 21: 0x28749b2b (stale) Mirror 25: 0x28749b2b (stale) Mirror 26: 0x28749b2b (stale) Mirror 27: 0x28749b2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29f5e): Mirror 2: 0x28749b2b (stale) Mirror 3: 0x28749b2b (stale) Mirror 5: 0x28749b2b (stale) Mirror 16: 0x28749b2b (stale) Mirror 18: 0x28749b2b (stale) Mirror 19: 0x28749b2b (stale) Mirror 20: 0x28749b2b (stale) Mirror 21: 0x28749b2b (stale) Mirror 25: 0x28749b2b (stale) Mirror 26: 0x28749b2b (stale) Mirror 27: 0x28749b2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29f5e: skipped 000988 1787787768.829022 mapread 0x00177e thru 0x0085ce (0x6e51 bytes) 000990 1787787768.869409 mapread 0x008f2b thru 0x013a7c (0xab52 bytes) 000992 1787787768.907393 read 0x026907 thru 0x029f5d (0x3657 bytes) truncating to largest ever: 0x3fea1 000993 1787787768.929426 trunc from 0x029f5e to 0x03fea0 (0x15f43 bytes) 000994 1787787768.961280 trunc from 0x03fea1 to 0x01a6de (0x257c2 bytes) 000995 1787787769.023408 mapread 0x017d2b thru 0x01a6de (0x29b4 bytes) 000996 1787787769.036319 mapwrite 0x019afe thru 0x01c20e (0x2711 bytes) 000997 1787787769.155287 mapread 0x012fd0 thru 0x01b226 (0x8257 bytes) 000998 1787787769.156448 trunc from 0x01c20f to 0x008e2e (0x133e0 bytes) 000999 1787787769.176953 read 0x006b1f thru 0x008e2e (0x2310 bytes) 001000 1787787769.177995 read 0x00197b thru 0x008e2e (0x74b4 bytes) All operations completed A-OK! PASS 70a (126s) == sanity-flr test 71: check mirror extend parent fid ==== 19:42:55 (1787787775) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 19:42:58 (1787787778) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.372584 s, 5.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185888 s, 5.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 19:43:08 (1787787788) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre2 Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5121248 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8499328 .. Extending file size to 5868160 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8006496 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8624288 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8799232 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8722944 .. Extending file size to 9097088 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58620 58622 58623 58624 58626 192.168.202.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre2 (opts:) 192.168.202.115@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:44:27 (1787787867) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre2 Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1390432 .. Extending file size to 1796096 .. Extending file size to 2455616 .. Extending file size to 7754368 .. Extending file size to 6179744 .. Extending file size to 7269760 .. Extending file size to 8412384 .. Extending file size to 8523552 .. Extending file size to 8630880 .. Extending file size to 8964416 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8635968 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8950816 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9082944 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 60277 60279 60280 60283 60284 done 192.168.202.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre2 (opts:) 192.168.202.115@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:45:47 (1787787947) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0556971 s, 37.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.452848 s, 4.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x633cf204 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:46:07 (1787787967) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0811997 s, 25.8 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 19:46:16 (1787787976) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d2:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:46:23 (1787787983) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.266889 s, 7.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:46:33 (1787787993) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.501013 s, 4.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:46:43 (1787788003) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.277521 s, 7.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:47:17 (1787788037) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg215-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939823 s, 11.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:48:21 (1787788101) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg215-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.292802 s, 7.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:49:21 (1787788161) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg215-server mds-ost sync done. Creating new pool test_204f oleg215-server: Pool lustre.test_204f created Adding targets to pool oleg215-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg215-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.400494 s, 5.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg215-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg215-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg215-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:50:41 (1787788241) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:50:48 (1787788248) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:50:54 (1787788254) oleg215-server: Pool lustre.test_206 created Adding targets to pool oleg215-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg215-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg215-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg215-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg215-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:51:17 (1787788277) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668461 s, 15.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584899 s, 17.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x4f3af74d Mirror 3: 0x4f3af74d PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:51:28 (1787788288) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:51:31 (1787788291) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:51:34 (1787788294) Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.105929 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xaa0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d20000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d20000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d20000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x10e0000 Read this (4 bytes): BBBB 192.168.202.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg215-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:52:09 (1787788329) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0141296 s, 580 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:52:17 (1787788337) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251958 s, 41.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f7 lmm_fid: [0x200000404:0x1f7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1f7 lmm_fid: [0x200000404:0x1f7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834423 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 19:52:19.000000000 -0400 Modify: 2026-08-26 19:52:19.000000000 -0400 Change: 2026-08-26 19:52:20.000000000 -0400 Birth: 2026-08-26 19:52:18.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:52:26 (1787788346) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:52:29 (1787788349) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.470982 s, 87.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0244776 s, 167 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 19:52 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f9:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 19:52 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:52:37 (1787788357) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:52:43 (1787788363) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1956 sec =========== 19:52:52 (1787788372) === sanity-flr: start cleanup 19:52:54 (1787788374) === === sanity-flr: finish cleanup 19:53:01 (1787788381) ===