-----============= acceptance-small: sanity-flr ============----- Thu Sep 17 15:33:27 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg250-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:33:47 (1789673627) === oleg250-client.virtnet: executing check_config_client /mnt/lustre oleg250-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg250-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff95d6d00e0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff95d6d00e0000.idle_timeout=debug disable quota as required oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:34:36 (1789673676) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7630] == sanity-flr test 0a: lfs mirror create with -N option == 15:34:39 (1789673679) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:34:58 (1789673698) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:35:02 (1789673702) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:35:06 (1789673706) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:35:35 (1789673735) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:35:38 (1789673738) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:35:42 (1789673742) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff95d6d00e0000.stats=clear osc.lustre-OST0001-osc-ffff95d6d00e0000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:35:51 (1789673751) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:36:04 (1789673764) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:36:13 (1789673773) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:36:17 (1789673777) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:36:27 (1789673787) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 15:36:36 (1789673796) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:36:44 (1789673804) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:36:46 (1789673806) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (10s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:36:56 (1789673816) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:37:08 (1789673828) PASS 7 (10s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:37:18 (1789673838) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 2.78129 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110996 s, 9.4 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:37:33 (1789673853) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392396 s, 26.7 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:37:43 (1789673863) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:37:55 (1789673875) Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95d6d00e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95d6d00e0000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg250-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95d6d00e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95d6d00e0000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (133s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:40:08 (1789674008) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0927513 s, 33.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95d6d00e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95d6d00e0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:40:50 (1789674050) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95d6d00e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95d6d00e0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95d6d00e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95d6d00e0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (87s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:42:17 (1789674137) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.74226 s, 9.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 17 15:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.115s user 0m0.002s sys 0m0.209s Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95d6d00e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95d6d00e0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.214s user 0m0.002s sys 0m0.189s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95d6d00e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95d6d00e0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.984s user 0m0.000s sys 0m0.189s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg250-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95d6d00e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95d6d00e0000.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:43:54 (1789674234) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:43:59 (1789674239) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:44:02 (1789674242) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 15:44:07 (1789674247) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 15:44:14 (1789674254) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.547183 s, 23.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:44:27 (1789674267) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483028 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017412 s, 60.2 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:44:33 (1789674273) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:44:41 (1789674281) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00143471 s, 714 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00151209 s, 677 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00163845 s, 625 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00188 s, 545 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 15:44:52 (1789674292) create test files with size 15748159 .. 0dbc29d87f522e3c2897815f0707d9df - 91a9039db4dfb772ffaa4aaad22a7a7c - 346f3181345d0d5413503cf130f2b661 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 15:45:21 (1789674321) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:45:36 (1789674336) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:45:43 (1789674343) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168797 s, 6.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263456 s, 39.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 15:45:54 (1789674354) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.196086 s, 10.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.591956 s, 7.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.392085 s, 5.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (21s) == sanity-flr test 42: lfs mirror verify ================= 15:46:16 (1789674376) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 15:46:20 (1789674380) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:46:23 (1789674383) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182734 s, 57.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 15:46:32 (1789674392) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:46:36 (1789674396) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:46:46 (1789674406) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.292728 s, 35.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:47:00 (1789674420) llite.lustre-ffff95d6e7221000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.251129 s, 41.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff95d6d17a6800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:47:15 (1789674435) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:48:00 (1789674480) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (46s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:48:47 (1789674527) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 15:48:51 (1789674531) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.400749 s, 7.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.323874 s, 9.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7780638d (nosync,stale) Mirror 2: 0x7780638d (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6ed2ab835a1f5ee3ef19071d5d11c790 - mirror 1 checksum: 3fb5876de04b6a7b462fe9956522e298 - mirror 2 checksum: 6ed2ab835a1f5ee3ef19071d5d11c790 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-17T15:48:55-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-17T15:48:55-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6ed2ab835a1f5ee3ef19071d5d11c790 - mirror 1 checksum: 3fb5876de04b6a7b462fe9956522e298 - mirror 2 checksum: 3fb5876de04b6a7b462fe9956522e298 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-17T15:48:55-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-17T15:48:55-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 48 (15s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:49:07 (1789674547) Failing mds1 on oleg250-server Stopping /mnt/lustre-mds1 (opts:) on oleg250-server 15:49:16 (1789674556) shut down facet: mds1 facet_host: oleg250-server facet_failover_host: oleg250-server Failover mds1 to oleg250-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 15:49:51 (1789674591) targets are mounted 15:49:51 (1789674591) facet_failover done oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478236 s, 21.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:50:14 (1789674614) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:50:18 (1789674618) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:50:23 (1789674623) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0155918 s, 1.1 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:50:33 (1789674633) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:50:37 (1789674637) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0104574 s, 392 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0774518 s, 52.9 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:50:46 (1789674646) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:51:13 (1789674673) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:51:45 (1789674705) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (28s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:52:13 (1789674733) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:52:40 (1789674760) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 15:52:48 (1789674768) mirror create pid 54365 mirror split pid 54367 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54365 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54367 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 15:53:57 (1789674837) Seed set to 4839 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002 1789674839.856341 MIRROR_EXTEND to 2 mirrors 000003 1789674840.662128 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000004 1789674840.822883 write 0x00ed4e thru 0x0159d4 (0x6c87 bytes) 000006 1789674840.824693 mapread 0x006b0f thru 0x012ab9 (0xbfab bytes) 000007 1789674840.837543 mapwrite 0x023737 thru 0x02eb6a (0xb434 bytes) 000008 1789674840.961914 mapread 0x01b496 thru 0x02a6f2 (0xf25d bytes) 000009 1789674840.972768 read 0x00dfd9 thru 0x0191c7 (0xb1ef bytes) 000011 1789674840.980123 MIRROR_EXTEND to 2 mirrors 000013 1789674842.273292 MIRROR_RESYNC 2 mirrors 000014 1789674842.883095 read 0x028841 thru 0x02eb6a (0x632a bytes) 000015 1789674842.912880 MIRROR_RESYNC 2 mirrors 000017 1789674843.339933 punch from 0x007211 to 0x00f6b3 (0x84a3 bytes) truncating to largest ever: 0x5c26 000021 1789674843.510206 trunc from 0x02eb6b to 0x005c25 (0x28f45 bytes) 000024 1789674843.575056 read 0x0042a9 thru 0x005c25 (0x197d bytes) 000026 1789674843.592852 read 0x0034a5 thru 0x005abc (0x2618 bytes) 000028 1789674843.604173 punch from 0x00362d to 0x005c25 (0x25f9 bytes) 000029 1789674843.620086 mapwrite 0x01ffd0 thru 0x02504f (0x5080 bytes) 000030 1789674843.787343 mapwrite 0x0382e4 thru 0x039778 (0x1495 bytes) 000032 1789674843.945549 mapwrite 0x0351d4 thru 0x03ffff (0xae2c bytes) 000033 1789674844.142296 write 0x0343f9 thru 0x03d0e5 (0x8ced bytes) 000035 1789674844.154902 read 0x0326d0 thru 0x03a45a (0x7d8b bytes) 000038 1789674844.164924 mapwrite 0x014396 thru 0x01dc7c (0x98e7 bytes) 000039 1789674844.280087 write 0x007906 thru 0x00a6d8 (0x2dd3 bytes) 000040 1789674844.282917 mapwrite 0x01d415 thru 0x02c4a2 (0xf08e bytes) 000041 1789674844.446545 mapread 0x017de0 thru 0x01cb2c (0x4d4d bytes) 000043 1789674844.457638 read 0x0356c7 thru 0x039abb (0x43f5 bytes) truncating to largest ever: 0x3a879 000044 1789674844.462438 trunc from 0x040000 to 0x03a878 (0x5787 bytes) 000048 1789674844.514308 trunc from 0x03a879 to 0x019c54 (0x20c24 bytes) 000049 1789674844.544618 write 0x01cf1e thru 0x022774 (0x5857 bytes) 000050 1789674844.552703 trunc from 0x022775 to 0x01085e (0x11f16 bytes) 000052 1789674844.581938 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1085f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1085f): Mirror 2: 0x8c037d2 (stale) CRC-32 checksum value for chunk [0, 0x1085f): Mirror 1: 0xdc7e2a89 Mirror 3: 0xdc7e2a89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1085f: skipped 000053 1789674846.031417 trunc from 0x01085f to 0x01d7bc (0xcf5e bytes) 000055 1789674846.095556 write 0x01d5d0 thru 0x020f46 (0x3977 bytes) 000056 1789674846.103579 write 0x0221dd thru 0x02f399 (0xd1bd bytes) 000057 1789674846.109556 punch from 0x007b92 to 0x00bfc7 (0x4436 bytes) 000058 1789674846.147524 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f39a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f39a): Mirror 2: 0xe3989b82 (stale) Mirror 3: 0xc2a91382 (stale) CRC-32 checksum value for chunk [0, 0x2f39a): Mirror 1: 0xa49726d5 Mirror 4: 0xa49726d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f39a: skipped 000059 1789674847.650631 mapread 0x0268b3 thru 0x0287d3 (0x1f21 bytes) 000060 1789674847.742958 read 0x005aaa thru 0x01267c (0xcbd3 bytes) 000061 1789674847.780050 read 0x0261db thru 0x02f399 (0x91bf bytes) 000062 1789674847.795982 punch from 0x0167d7 to 0x02101e (0xa848 bytes) 000063 1789674847.948582 mapread 0x002571 thru 0x002a67 (0x4f7 bytes) 000064 1789674847.964566 trunc from 0x02f39a to 0x02d98e (0x1a0b bytes) 000066 1789674848.008311 mapwrite 0x00f88f thru 0x018cb2 (0x9424 bytes) 000067 1789674848.096287 write 0x031f6d thru 0x03ffff (0xe093 bytes) 000068 1789674848.099774 mapread 0x01b8ea thru 0x01c46a (0xb81 bytes) 000069 1789674848.109313 MIRROR_RESYNC 4 mirrors 000070 1789674849.251036 read 0x03dc0b thru 0x03ffff (0x23f5 bytes) 000071 1789674849.267802 mapwrite 0x0233b8 thru 0x02e161 (0xadaa bytes) 000072 1789674849.706015 mapread 0x0087f9 thru 0x016257 (0xda5f bytes) 000074 1789674849.772726 mapwrite 0x00eeb0 thru 0x019b9b (0xacec bytes) 000075 1789674849.913065 read 0x02697b thru 0x0311e6 (0xa86c bytes) truncating to largest ever: 0x3c85b 000076 1789674849.913760 trunc from 0x040000 to 0x03c85a (0x37a5 bytes) 000079 1789674849.991978 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c85b): Mirror 2: 0xc6dde15 (stale) Mirror 3: 0xc6dde15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c85b): Mirror 2: 0xc6dde15 (stale) Mirror 3: 0xc6dde15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c85b: skipped 000080 1789674850.332355 read 0x0164f1 thru 0x020f04 (0xaa14 bytes) 000081 1789674850.342300 write 0x00b515 thru 0x0134e5 (0x7fd1 bytes) 000082 1789674850.352324 write 0x0156c2 thru 0x021a00 (0xc33f bytes) 000085 1789674850.367052 mapwrite 0x017a7b thru 0x01cca7 (0x522d bytes) 000086 1789674850.431627 read 0x031534 thru 0x03c85a (0xb327 bytes) 000087 1789674850.442209 mapread 0x033ac0 thru 0x035a1c (0x1f5d bytes) 000088 1789674850.443872 punch from 0x0240db to 0x029848 (0x576e bytes) 000089 1789674850.448998 mapwrite 0x008c12 thru 0x00963a (0xa29 bytes) 000091 1789674850.498051 write 0x01b5ff thru 0x0216b9 (0x60bb bytes) 000092 1789674850.505154 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c85b): Mirror 3: 0xc6dde15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c85b): Mirror 3: 0xc6dde15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c85b: skipped 000093 1789674851.015405 mapwrite 0x0057ca thru 0x013e2f (0xe666 bytes) 000095 1789674851.194492 read 0x02015b thru 0x0271bb (0x7061 bytes) 000098 1789674851.226638 write 0x01e38f thru 0x025949 (0x75bb bytes) 000101 1789674851.242241 mapread 0x00061d thru 0x010467 (0xfe4b bytes) 000103 1789674851.274448 mapread 0x00bbcd thru 0x00f3e3 (0x3817 bytes) 000104 1789674851.277250 mapwrite 0x0368ae thru 0x03ffff (0x9752 bytes) 000105 1789674851.447965 mapread 0x037c8b thru 0x03ffff (0x8375 bytes) 000106 1789674851.478078 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4e4265fd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe98265e1 Mirror 4: 0xe98265e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000107 1789674852.343414 read 0x002b13 thru 0x011ce3 (0xf1d1 bytes) 000108 1789674852.375227 mapwrite 0x00db6c thru 0x018054 (0xa4e9 bytes) 000109 1789674852.707510 read 0x02cba4 thru 0x02d4c5 (0x922 bytes) 000111 1789674852.724785 mapwrite 0x0147d0 thru 0x0214ea (0xcd1b bytes) 000113 1789674852.841151 mapwrite 0x01e566 thru 0x0287a9 (0xa244 bytes) 000115 1789674852.897116 punch from 0x0172f9 to 0x018fe2 (0x1cea bytes) 000116 1789674852.907480 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4e4265fd (stale) Mirror 4: 0xe98265e1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x788deb6 Mirror 5: 0x788deb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000117 1789674854.166963 trunc from 0x040000 to 0x01be7d (0x24182 bytes) 000118 1789674854.286985 trunc from 0x01be7e to 0x018159 (0x3d24 bytes) 000119 1789674854.310005 trunc from 0x01815a to 0x018381 (0x228 bytes) 000121 1789674854.325848 read 0x0015de thru 0x004a4b (0x346e bytes) 000122 1789674854.333700 trunc from 0x018382 to 0x009541 (0xee40 bytes) 000123 1789674854.355308 write 0x007c71 thru 0x00e6c6 (0x6a56 bytes) 000124 1789674854.371828 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe6c7): Mirror 4: 0x73f025b3 (stale) Mirror 5: 0x1df24987 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe6c7): Mirror 4: 0x73f025b3 (stale) Mirror 5: 0x1df24987 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe6c7: skipped 000126 1789674854.698293 read 0x00d858 thru 0x00e6c6 (0xe6f bytes) 000127 1789674854.704625 write 0x01809b thru 0x01cfc8 (0x4f2e bytes) 000128 1789674854.706217 read 0x01ae6c thru 0x01cfc8 (0x215d bytes) 000129 1789674854.706883 read 0x001f6d thru 0x0066f9 (0x478d bytes) 000131 1789674854.729186 trunc from 0x01cfc9 to 0x00e452 (0xeb76 bytes) 000132 1789674854.758004 write 0x0349da thru 0x03ffff (0xb626 bytes) 000136 1789674854.762294 punch from 0x0374fe to 0x03ffff (0x8b02 bytes) 000137 1789674854.776920 punch from 0x00b24b to 0x00d696 (0x244c bytes) 000138 1789674854.785634 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe98265e1 (stale) Mirror 5: 0x788deb6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9fd59499 Mirror 6: 0x9fd59499 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000139 1789674856.365019 trunc from 0x040000 to 0x00e1a5 (0x31e5a bytes) 000141 1789674856.559334 read 0x0019ce thru 0x00e1a5 (0xc7d8 bytes) 000142 1789674856.574973 punch from 0x00bfbd to 0x00e1a5 (0x21e9 bytes) 000144 1789674856.589822 MIRROR_RESYNC 4 mirrors 000145 1789674858.019863 MIRROR_EXTEND to 5 mirrors 000146 1789674858.817097 trunc from 0x00e1a6 to 0x0064ed (0x7cb8 bytes) 000147 1789674858.972864 read 0x001ecd thru 0x0064ed (0x4621 bytes) 000148 1789674859.009291 trunc from 0x0064ee to 0x02eea1 (0x289b4 bytes) 000149 1789674859.040100 mapread 0x0180c0 thru 0x0262df (0xe220 bytes) 000151 1789674859.122846 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2eea2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eea2): Mirror 4: 0xc1144706 (stale) Mirror 5: 0xc1144706 (stale) Mirror 6: 0xc1144706 (stale) Mirror 7: 0xc1144706 (stale) CRC-32 checksum value for chunk [0, 0x2eea2): Mirror 1: 0x923cd92e Mirror 8: 0x923cd92e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eea2: skipped 000152 1789674860.886990 trunc from 0x02eea2 to 0x032648 (0x37a7 bytes) 000153 1789674861.049803 punch from 0x0110c1 to 0x01cb07 (0xba47 bytes) 000154 1789674861.055848 trunc from 0x032649 to 0x00d695 (0x24fb3 bytes) 000157 1789674861.079741 punch from 0x008e09 to 0x009795 (0x98d bytes) 000159 1789674861.095222 mapwrite 0x008603 thru 0x013e4e (0xb84c bytes) 000160 1789674861.260191 write 0x039a78 thru 0x03ffff (0x6588 bytes) 000162 1789674861.263960 punch from 0x00807b to 0x012da7 (0xad2d bytes) 000164 1789674861.302827 mapread 0x02f364 thru 0x035b6d (0x680a bytes) 000168 1789674861.318353 trunc from 0x040000 to 0x014ea3 (0x2b15c bytes) 000169 1789674861.355236 mapread 0x00fc6e thru 0x01136f (0x1702 bytes) 000170 1789674861.374568 trunc from 0x014ea4 to 0x0255e8 (0x10745 bytes) 000172 1789674861.414938 mapwrite 0x003d31 thru 0x0072cb (0x359b bytes) 000173 1789674861.527811 write 0x018a04 thru 0x01fbb8 (0x71b5 bytes) 000174 1789674861.553386 write 0x002c39 thru 0x0080d8 (0x54a0 bytes) truncating to largest ever: 0x3dafd 000175 1789674861.580989 trunc from 0x0255e9 to 0x03dafc (0x18514 bytes) 000176 1789674861.638632 punch from 0x02c6be to 0x03a77c (0xe0bf bytes) 000177 1789674861.702503 mapwrite 0x016816 thru 0x01e433 (0x7c1e bytes) 000179 1789674861.847731 write 0x00c92e thru 0x01851f (0xbbf2 bytes) 000180 1789674861.853612 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dafd): Mirror 5: 0x708b194d (stale) Mirror 6: 0x708b194d (stale) Mirror 7: 0x708b194d (stale) Mirror 8: 0xc025bd55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dafd): Mirror 5: 0x708b194d (stale) Mirror 6: 0x708b194d (stale) Mirror 7: 0x708b194d (stale) Mirror 8: 0xc025bd55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dafd: skipped 000181 1789674862.554271 mapread 0x030ed2 thru 0x03a6c5 (0x97f4 bytes) 000182 1789674862.573644 trunc from 0x03dafd to 0x00f161 (0x2e99b bytes) 000183 1789674862.598632 trunc from 0x00f162 to 0x021468 (0x12307 bytes) 000184 1789674862.618257 write 0x0212ba thru 0x02e68b (0xd3d2 bytes) 000186 1789674862.626929 punch from 0x008efc to 0x00bd60 (0x2e65 bytes) 000187 1789674862.648652 mapwrite 0x013e69 thru 0x01980b (0x59a3 bytes) 000188 1789674862.774612 mapwrite 0x006293 thru 0x011887 (0xb5f5 bytes) 000189 1789674862.837304 mapread 0x0288e4 thru 0x02e68b (0x5da8 bytes) 000193 1789674862.845876 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e68c): Mirror 5: 0x3685db42 (stale) Mirror 6: 0x3685db42 (stale) Mirror 8: 0x4d724ff1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e68c): Mirror 5: 0x3685db42 (stale) Mirror 6: 0x3685db42 (stale) Mirror 8: 0x4d724ff1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e68c: skipped 000194 1789674863.232091 trunc from 0x02e68c to 0x00404a (0x2a641 bytes) 000195 1789674863.252588 write 0x02f824 thru 0x03e999 (0xf176 bytes) 000196 1789674863.254591 write 0x03f1df thru 0x03ffff (0xe21 bytes) 000197 1789674863.257262 read 0x02c85b thru 0x03ab96 (0xe33c bytes) 000198 1789674863.262496 mapwrite 0x00b6ff thru 0x00f50a (0x3e0c bytes) 000199 1789674863.345167 trunc from 0x040000 to 0x01d6a3 (0x2295c bytes) 000200 1789674863.376852 write 0x016492 thru 0x02622e (0xfd9d bytes) 000201 1789674863.386706 read 0x017aca thru 0x021f5b (0xa492 bytes) 000202 1789674863.387618 mapwrite 0x0013da thru 0x004a94 (0x36bb bytes) 000204 1789674863.459557 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2622f): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2622f): Mirror 5: 0xe0a0faf0 (stale) Mirror 6: 0xe0a0faf0 (stale) Mirror 8: 0xd220f230 (stale) CRC-32 checksum value for chunk [0, 0x2622f): Mirror 1: 0xd8ab7df Mirror 9: 0xd8ab7df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2622f: skipped 000205 1789674864.748169 punch from 0x01c238 to 0x01f0f0 (0x2eb9 bytes) 000207 1789674864.862125 read 0x0119d5 thru 0x01b105 (0x9731 bytes) 000208 1789674864.872230 write 0x0253e7 thru 0x02fcc2 (0xa8dc bytes) 000209 1789674864.878226 trunc from 0x02fcc3 to 0x02ee26 (0xe9c bytes) 000210 1789674864.903883 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ee27): Mirror 5: 0xbcbea4a8 (stale) Mirror 8: 0x62aab006 (stale) Mirror 9: 0xd4831022 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee27): Mirror 5: 0xbcbea4a8 (stale) Mirror 8: 0x62aab006 (stale) Mirror 9: 0xd4831022 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee27: skipped 000211 1789674865.260136 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ee27): Mirror 5: 0xbcbea4a8 (stale) Mirror 9: 0xd4831022 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee27): Mirror 5: 0xbcbea4a8 (stale) Mirror 9: 0xd4831022 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee27: skipped 000212 1789674865.573406 write 0x00fbde thru 0x01a48e (0xa8b1 bytes) 000213 1789674865.585710 mapwrite 0x02ef8a thru 0x0392ef (0xa366 bytes) 000217 1789674865.728061 punch from 0x000ea8 to 0x000f18 (0x71 bytes) 000218 1789674865.742795 read 0x00d7f2 thru 0x0148de (0x70ed bytes) 000219 1789674865.751147 punch from 0x00ecb8 to 0x0103b8 (0x1701 bytes) truncating to largest ever: 0x3ded0 000220 1789674865.759565 trunc from 0x0392f0 to 0x03decf (0x4be0 bytes) 000221 1789674865.788835 trunc from 0x03ded0 to 0x02f9af (0xe520 bytes) 000222 1789674865.814904 trunc from 0x02f9b0 to 0x003b65 (0x2be4a bytes) 000225 1789674865.857105 MIRROR_RESYNC 3 mirrors 000227 1789674867.019245 trunc from 0x003b66 to 0x00c121 (0x85bc bytes) 000228 1789674867.216235 MIRROR_RESYNC 3 mirrors 000230 1789674868.334326 MIRROR_RESYNC 3 mirrors 000233 1789674868.769314 write 0x03a6c9 thru 0x03ffff (0x5937 bytes) 000234 1789674868.874303 mapread 0x02bfbe thru 0x02e108 (0x214b bytes) 000235 1789674868.898864 write 0x01eaa0 thru 0x021ce4 (0x3245 bytes) 000236 1789674868.914678 write 0x02690b thru 0x033237 (0xc92d bytes) 000237 1789674868.918394 read 0x03c475 thru 0x03ffff (0x3b8b bytes) 000238 1789674868.919502 write 0x0093ab thru 0x0148b1 (0xb507 bytes) 000241 1789674868.931772 mapread 0x030af0 thru 0x03a2c4 (0x97d5 bytes) 000244 1789674868.940139 mapwrite 0x02fbb5 thru 0x039ebf (0xa30b bytes) 000245 1789674869.019137 mapwrite 0x01c4a0 thru 0x02b275 (0xedd6 bytes) 000246 1789674869.116903 punch from 0x009656 to 0x0107fb (0x71a6 bytes) 000247 1789674869.122246 trunc from 0x040000 to 0x01d767 (0x22898 bytes) 000250 1789674869.161315 punch from 0x004571 to 0x00a61c (0x60ac bytes) 000251 1789674869.167143 trunc from 0x01d768 to 0x01b964 (0x1e03 bytes) 000252 1789674869.189928 MIRROR_RESYNC 3 mirrors 000253 1789674870.357811 MIRROR_EXTEND to 4 mirrors 000254 1789674871.577702 punch from 0x000a57 to 0x00139e (0x948 bytes) 000255 1789674871.694977 write 0x0061d0 thru 0x00bff3 (0x5e24 bytes) 000256 1789674871.715280 mapread 0x0199e7 thru 0x01b964 (0x1f7e bytes) 000257 1789674871.755765 punch from 0x018081 to 0x01b964 (0x38e4 bytes) 000258 1789674871.800756 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b965): Mirror 5: 0xb55089cc (stale) Mirror 9: 0xb55089cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b965): Mirror 5: 0xb55089cc (stale) Mirror 9: 0xb55089cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b965: skipped 000259 1789674872.427973 trunc from 0x01b965 to 0x023e8d (0x8529 bytes) 000261 1789674872.448157 mapread 0x014a1a thru 0x023e8d (0xf474 bytes) 000262 1789674872.484181 trunc from 0x023e8e to 0x032ffa (0xf16d bytes) 000263 1789674872.508101 mapwrite 0x0339fa thru 0x03ef24 (0xb52b bytes) 000266 1789674872.657302 punch from 0x038775 to 0x03ef24 (0x67b0 bytes) 000269 1789674872.677571 write 0x02fd7c thru 0x031f92 (0x2217 bytes) 000271 1789674872.682556 write 0x02fc52 thru 0x032843 (0x2bf2 bytes) 000272 1789674872.685597 read 0x00c9e2 thru 0x013960 (0x6f7f bytes) 000275 1789674872.694336 write 0x01c9d4 thru 0x0226d6 (0x5d03 bytes) 000276 1789674872.696139 mapwrite 0x019348 thru 0x01f9a6 (0x665f bytes) 000277 1789674872.763849 trunc from 0x03ef25 to 0x029094 (0x15e90 bytes) 000278 1789674872.781406 read 0x01e1ab thru 0x029094 (0xaeea bytes) 000279 1789674872.796549 write 0x0234af thru 0x026082 (0x2bd4 bytes) 000280 1789674872.798619 mapread 0x01933d thru 0x0254b5 (0xc179 bytes) 000281 1789674872.799719 mapwrite 0x027654 thru 0x02c85d (0x520a bytes) 000282 1789674872.873747 read 0x00241c thru 0x00f048 (0xcc2d bytes) 000283 1789674872.875120 read 0x00d74f thru 0x00fee6 (0x2798 bytes) 000284 1789674872.875836 mapwrite 0x0078a6 thru 0x00e58a (0x6ce5 bytes) 000286 1789674872.931625 mapread 0x020262 thru 0x02c85d (0xc5fc bytes) 000287 1789674872.932633 mapwrite 0x03857f thru 0x03ffff (0x7a81 bytes) 000288 1789674873.037424 mapread 0x00748e thru 0x015b59 (0xe6cc bytes) 000289 1789674873.038598 trunc from 0x040000 to 0x01a973 (0x2568c bytes) 000290 1789674873.062672 mapread 0x01767f thru 0x01a973 (0x32f5 bytes) 000291 1789674873.063601 punch from 0x017ff5 to 0x01a472 (0x247e bytes) 000292 1789674873.077114 mapwrite 0x028a9f thru 0x02dd2c (0x528e bytes) 000293 1789674873.203838 mapwrite 0x02e2b8 thru 0x03d412 (0xf15b bytes) 000294 1789674873.501293 punch from 0x01de09 to 0x025b9b (0x7d93 bytes) 000297 1789674873.514670 read 0x01f77c thru 0x028ff9 (0x987e bytes) 000298 1789674873.532860 punch from 0x01cf8b to 0x01ee8a (0x1f00 bytes) 000299 1789674873.540698 mapread 0x03654f thru 0x03af3c (0x49ee bytes) 000302 1789674873.546276 mapwrite 0x03efdd thru 0x03ffff (0x1023 bytes) 000303 1789674873.681918 mapwrite 0x00835a thru 0x00a455 (0x20fc bytes) 000304 1789674873.745153 mapread 0x03fd4a thru 0x03ffff (0x2b6 bytes) 000308 1789674873.752170 punch from 0x01531c to 0x0181b1 (0x2e96 bytes) 000310 1789674873.765084 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1da747d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1da747d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000311 1789674874.228116 mapread 0x00b65a thru 0x00cced (0x1694 bytes) 000312 1789674874.244652 read 0x028264 thru 0x03333e (0xb0db bytes) 000313 1789674874.256708 punch from 0x00a565 to 0x00e377 (0x3e13 bytes) 000314 1789674874.262233 mapwrite 0x03ccae thru 0x03ffff (0x3352 bytes) 000316 1789674874.363266 punch from 0x012ad7 to 0x01e7ee (0xbd18 bytes) 000318 1789674874.373059 write 0x018d51 thru 0x025caa (0xcf5a bytes) 000321 1789674874.391740 mapwrite 0x02622e thru 0x02e748 (0x851b bytes) 000324 1789674874.527503 mapwrite 0x033085 thru 0x03f599 (0xc515 bytes) 000326 1789674874.651642 mapread 0x003b41 thru 0x01046f (0xc92f bytes) 000330 1789674874.685172 punch from 0x0011d0 to 0x004a81 (0x38b2 bytes) 000331 1789674874.693911 write 0x024332 thru 0x028ed0 (0x4b9f bytes) 000332 1789674874.717398 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000338 1789674874.987226 trunc from 0x040000 to 0x033331 (0xccce bytes) 000344 1789674875.047517 mapread 0x00d687 thru 0x01537b (0x7cf5 bytes) 000345 1789674875.095200 MIRROR_EXTEND to 2 mirrors 000346 1789674876.346908 MIRROR_EXTEND to 3 mirrors 000348 1789674877.769666 trunc from 0x033332 to 0x0169a4 (0x1c98d bytes) 000349 1789674877.945014 write 0x00cc46 thru 0x01a2ba (0xd675 bytes) 000350 1789674877.958291 mapread 0x004c46 thru 0x005d44 (0x10ff bytes) 000351 1789674877.980938 mapread 0x00dce0 thru 0x012dba (0x50db bytes) 000352 1789674877.983610 trunc from 0x01a2bb to 0x02c15f (0x11ea5 bytes) 000353 1789674878.064113 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2c160): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c160): Mirror 2: 0xde2fa97e (stale) Mirror 3: 0xde2fa97e (stale) CRC-32 checksum value for chunk [0, 0x2c160): Mirror 1: 0xc3a4451d Mirror 4: 0xc3a4451d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c160: skipped 000355 1789674880.459960 MIRROR_RESYNC 4 mirrors 000356 1789674881.444909 punch from 0x026edc to 0x02c15f (0x5284 bytes) 000357 1789674881.518689 punch from 0x020920 to 0x02be88 (0xb569 bytes) 000359 1789674881.524971 punch from 0x029e5a to 0x02b6f6 (0x189d bytes) 000362 1789674881.531680 read 0x01d9fe thru 0x023d9d (0x63a0 bytes) 000363 1789674881.545850 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c160): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c160): Mirror 2: 0xc3a4451d (stale) Mirror 3: 0xc3a4451d (stale) Mirror 4: 0xc3a4451d (stale) CRC-32 checksum value for chunk [0, 0x2c160): Mirror 1: 0xc3a4451d Mirror 5: 0xc3a4451d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c160: skipped 000364 1789674882.858176 punch from 0x021388 to 0x02c15f (0xadd8 bytes) 000365 1789674882.989281 write 0x03cb61 thru 0x03e4ee (0x198e bytes) 000366 1789674883.002115 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e4ef): Mirror 2: 0xd1919b84 (stale) Mirror 3: 0xd1919b84 (stale) Mirror 4: 0xd1919b84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4ef): Mirror 2: 0xd1919b84 (stale) Mirror 3: 0xd1919b84 (stale) Mirror 4: 0xd1919b84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4ef: skipped 000368 1789674883.454910 read 0x013f16 thru 0x01ba17 (0x7b02 bytes) 000370 1789674883.475264 punch from 0x03becf to 0x03e354 (0x2486 bytes) 000371 1789674883.485187 trunc from 0x03e4ef to 0x0027d2 (0x3bd1c bytes) 000372 1789674883.510606 trunc from 0x0027d3 to 0x009e3c (0x766a bytes) 000373 1789674883.528896 write 0x02b2be thru 0x03b2b4 (0xfff7 bytes) 000376 1789674883.533139 punch from 0x02e17c to 0x037dde (0x9c63 bytes) 000377 1789674883.547532 read 0x0344ec thru 0x039731 (0x5246 bytes) 000378 1789674883.561715 mapread 0x012ef6 thru 0x015329 (0x2434 bytes) 000379 1789674883.563086 read 0x0263a4 thru 0x0356a8 (0xf305 bytes) 000381 1789674883.564681 trunc from 0x03b2b5 to 0x0135ac (0x27d08 bytes) 000382 1789674883.598586 mapread 0x00f7ba thru 0x0135ac (0x3df3 bytes) 000383 1789674883.599698 read 0x013049 thru 0x0135ac (0x564 bytes) 000384 1789674883.600640 mapwrite 0x039cae thru 0x03a741 (0xa94 bytes) 000385 1789674883.710985 trunc from 0x03a742 to 0x0254f5 (0x1524c bytes) 000386 1789674883.733622 mapread 0x0002a7 thru 0x008903 (0x865d bytes) 000387 1789674883.735738 punch from 0x01bb1d to 0x0254f5 (0x99d9 bytes) 000391 1789674883.746748 mapwrite 0x021980 thru 0x025534 (0x3bb5 bytes) 000393 1789674883.892900 mapwrite 0x0122cf thru 0x018e20 (0x6b52 bytes) 000394 1789674883.965427 MIRROR_RESYNC 4 mirrors 000395 1789674885.366250 punch from 0x008b78 to 0x01597c (0xce05 bytes) 000396 1789674885.428743 read 0x00851f thru 0x01717a (0xec5c bytes) 000398 1789674885.439356 read 0x016940 thru 0x01b360 (0x4a21 bytes) 000399 1789674885.446583 trunc from 0x025535 to 0x029ddd (0x48a9 bytes) 000401 1789674885.472906 read 0x0014ce thru 0x00c47d (0xafb0 bytes) 000402 1789674885.473425 read 0x005717 thru 0x00dc29 (0x8513 bytes) 000403 1789674885.478593 MIRROR_RESYNC 4 mirrors 000404 1789674886.792666 write 0x01df3f thru 0x0207de (0x28a0 bytes) 000405 1789674886.882818 mapread 0x008199 thru 0x0158d2 (0xd73a bytes) 000406 1789674886.915139 mapread 0x00b85d thru 0x019352 (0xdaf6 bytes) 000407 1789674886.918448 write 0x02d5ae thru 0x0356fa (0x814d bytes) 000408 1789674886.920890 write 0x011765 thru 0x01bed8 (0xa774 bytes) 000409 1789674886.924152 read 0x023782 thru 0x02f207 (0xba86 bytes) 000410 1789674886.937509 read 0x001102 thru 0x00b93a (0xa839 bytes) 000412 1789674886.951402 trunc from 0x0356fb to 0x01a594 (0x1b166 bytes) 000414 1789674886.982663 trunc from 0x01a595 to 0x0181a2 (0x23f2 bytes) 000416 1789674887.015956 mapread 0x000f85 thru 0x00fc66 (0xece2 bytes) 000417 1789674887.019718 mapwrite 0x019d55 thru 0x01cf42 (0x31ee bytes) 000418 1789674887.136514 trunc from 0x01cf43 to 0x009d0e (0x13234 bytes) 000419 1789674887.176010 trunc from 0x009d0f to 0x032458 (0x2874a bytes) 000420 1789674887.210262 mapread 0x018366 thru 0x0203ab (0x8046 bytes) 000422 1789674887.270888 trunc from 0x032459 to 0x01f6d4 (0x12d84 bytes) 000423 1789674887.298594 punch from 0x01ceca to 0x01eff8 (0x212f bytes) 000425 1789674887.311498 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f6d5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f6d5): Mirror 2: 0xafa3215 (stale) Mirror 3: 0xafa3215 (stale) Mirror 4: 0xafa3215 (stale) CRC-32 checksum value for chunk [0, 0x1f6d5): Mirror 1: 0x754c096b Mirror 5: 0x754c096b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f6d5: skipped 000426 1789674888.522236 mapread 0x019e55 thru 0x01ae3b (0xfe7 bytes) 000428 1789674888.582567 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1f6d5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f6d5): Mirror 2: 0xafa3215 (stale) Mirror 3: 0xafa3215 (stale) Mirror 4: 0xafa3215 (stale) CRC-32 checksum value for chunk [0, 0x1f6d5): Mirror 1: 0x754c096b Mirror 5: 0x754c096b Mirror 6: 0x754c096b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f6d5: skipped 000430 1789674890.198501 punch from 0x01152a to 0x012155 (0xc2c bytes) 000432 1789674890.333132 punch from 0x00a7b0 to 0x010a58 (0x62a9 bytes) 000434 1789674890.340607 mapwrite 0x026b1b thru 0x02acb3 (0x4199 bytes) 000435 1789674890.475527 punch from 0x021998 to 0x02acb3 (0x931c bytes) 000437 1789674890.489442 punch from 0x017a66 to 0x01f65d (0x7bf8 bytes) 000438 1789674890.495234 write 0x01b2ce thru 0x02b190 (0xfec3 bytes) 000439 1789674890.503104 mapwrite 0x02ad29 thru 0x0340ad (0x9385 bytes) 000440 1789674890.589581 write 0x00db61 thru 0x01d8eb (0xfd8b bytes) 000441 1789674890.598712 write 0x017371 thru 0x026cab (0xf93b bytes) 000442 1789674890.601055 mapread 0x02ac0c thru 0x031305 (0x66fa bytes) 000444 1789674890.603051 mapwrite 0x007b8a thru 0x0128ac (0xad23 bytes) 000446 1789674890.674898 trunc from 0x0340ae to 0x01126a (0x22e43 bytes) 000447 1789674890.691853 mapwrite 0x0026ac thru 0x00aada (0x842f bytes) 000448 1789674890.739041 write 0x03a52c thru 0x03ffff (0x5ad4 bytes) 000450 1789674890.744037 write 0x005a64 thru 0x00f011 (0x95ae bytes) 000454 1789674890.753318 MIRROR_RESYNC 6 mirrors 000455 1789674892.613634 trunc from 0x040000 to 0x015b80 (0x2a47f bytes) 000457 1789674892.726574 read 0x00c17d thru 0x015b80 (0x9a04 bytes) 000458 1789674892.747865 mapread 0x002700 thru 0x00e4bb (0xbdbc bytes) 000459 1789674892.797541 punch from 0x006ff8 to 0x009102 (0x210b bytes) 000460 1789674892.815013 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x15b81): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15b81): Mirror 2: 0x5506badf (stale) Mirror 3: 0x5506badf (stale) Mirror 4: 0x5506badf (stale) Mirror 5: 0x5506badf (stale) Mirror 6: 0x5506badf (stale) CRC-32 checksum value for chunk [0, 0x15b81): Mirror 1: 0x3aa5d01 Mirror 7: 0x3aa5d01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15b81: skipped 000461 1789674894.209100 trunc from 0x015b81 to 0x01402a (0x1b56 bytes) 000464 1789674894.315349 write 0x02fea7 thru 0x03850e (0x8668 bytes) 000465 1789674894.318257 read 0x0328c1 thru 0x03850e (0x5c4e bytes) 000467 1789674894.320191 mapread 0x005971 thru 0x012b44 (0xd1d4 bytes) 000469 1789674894.358527 read 0x007fdc thru 0x00ae4d (0x2e72 bytes) 000470 1789674894.365665 MIRROR_RESYNC 7 mirrors 000471 1789674897.237828 MIRROR_EXTEND to 8 mirrors 000472 1789674899.043907 trunc from 0x03850f to 0x02b5d2 (0xcf3c bytes) 000476 1789674899.177449 mapwrite 0x007d93 thru 0x00e3a0 (0x660e bytes) 000477 1789674899.263317 mapread 0x01c99a thru 0x01f35a (0x29c1 bytes) 000478 1789674899.276802 mapread 0x00316c thru 0x0125fa (0xf48f bytes) 000480 1789674899.288332 write 0x03f214 thru 0x03ffff (0xdec bytes) 000481 1789674899.290811 read 0x0051e5 thru 0x007cf3 (0x2b0f bytes) 000482 1789674899.291804 write 0x020f03 thru 0x0305e1 (0xf6df bytes) 000484 1789674899.301512 read 0x00df52 thru 0x0162ca (0x8379 bytes) 000486 1789674899.303350 punch from 0x02aae7 to 0x030744 (0x5c5e bytes) 000487 1789674899.331544 MIRROR_RESYNC 8 mirrors 000491 1789674901.523289 trunc from 0x040000 to 0x00552d (0x3aad2 bytes) 000492 1789674901.593761 write 0x008799 thru 0x00fba4 (0x740c bytes) 000494 1789674901.600001 read 0x006059 thru 0x00b9a2 (0x594a bytes) 000495 1789674901.606872 punch from 0x00197e to 0x00a68f (0x8d12 bytes) 000496 1789674901.620989 write 0x0208cc thru 0x022064 (0x1799 bytes) 000497 1789674901.622215 mapread 0x00e862 thru 0x0126c8 (0x3e67 bytes) 000498 1789674901.640498 mapwrite 0x028446 thru 0x033af1 (0xb6ac bytes) 000499 1789674901.717979 trunc from 0x033af2 to 0x00ffe9 (0x23b08 bytes) 000500 1789674901.738021 mapread 0x002b5c thru 0x00ffe9 (0xd48e bytes) 000503 1789674901.749287 punch from 0x00e154 to 0x00f28d (0x113a bytes) 000505 1789674901.760169 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xffea): Mirror 3: 0xd98b717a (stale) Mirror 4: 0xd98b717a (stale) Mirror 5: 0xd98b717a (stale) Mirror 6: 0xd98b717a (stale) Mirror 7: 0xd98b717a (stale) Mirror 8: 0xd98b717a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xffea): Mirror 3: 0xd98b717a (stale) Mirror 4: 0xd98b717a (stale) Mirror 5: 0xd98b717a (stale) Mirror 6: 0xd98b717a (stale) Mirror 7: 0xd98b717a (stale) Mirror 8: 0xd98b717a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xffea: skipped 000506 1789674902.181579 mapwrite 0x00a18d thru 0x00ff97 (0x5e0b bytes) 000507 1789674902.251968 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xffea): Mirror 4: 0xd98b717a (stale) Mirror 5: 0xd98b717a (stale) Mirror 6: 0xd98b717a (stale) Mirror 7: 0xd98b717a (stale) Mirror 8: 0xd98b717a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xffea): Mirror 4: 0xd98b717a (stale) Mirror 5: 0xd98b717a (stale) Mirror 6: 0xd98b717a (stale) Mirror 7: 0xd98b717a (stale) Mirror 8: 0xd98b717a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xffea: skipped 000508 1789674902.706968 read 0x0027fe thru 0x00ffe9 (0xd7ec bytes) 000510 1789674902.716951 trunc from 0x00ffea to 0x00c288 (0x3d61 bytes) 000511 1789674902.742555 trunc from 0x00c289 to 0x0048b1 (0x79d7 bytes) 000513 1789674902.763925 punch from 0x004824 to 0x0048b1 (0x8e bytes) 000514 1789674902.770569 mapwrite 0x0264a3 thru 0x032e90 (0xc9ee bytes) 000516 1789674902.877879 trunc from 0x032e91 to 0x01bb27 (0x17369 bytes) 000518 1789674902.911934 MIRROR_RESYNC 6 mirrors 000519 1789674904.381782 MIRROR_RESYNC 6 mirrors 000520 1789674904.902985 write 0x03feff thru 0x03ffff (0x101 bytes) 000521 1789674904.967524 mapread 0x024d3a thru 0x02ad2b (0x5ff2 bytes) 000522 1789674904.986889 punch from 0x00ee28 to 0x016b7b (0x7d54 bytes) 000523 1789674905.001349 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x24d39cb4 Mirror 9: 0x24d39cb4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000524 1789674907.054760 punch from 0x003f30 to 0x011e08 (0xded9 bytes) 000526 1789674907.187907 mapread 0x036cdd thru 0x03a5d4 (0x38f8 bytes) 000527 1789674907.234089 write 0x03cbef thru 0x03ffff (0x3411 bytes) 000529 1789674907.235709 mapwrite 0x00f93d thru 0x012617 (0x2cdb bytes) 000530 1789674907.311232 trunc from 0x040000 to 0x00309f (0x3cf60 bytes) 000532 1789674907.340615 read 0x000248 thru 0x00309f (0x2e58 bytes) 000533 1789674907.347585 trunc from 0x0030a0 to 0x0358b5 (0x32816 bytes) 000534 1789674907.366076 write 0x021648 thru 0x028b41 (0x74fa bytes) 000535 1789674907.377194 read 0x022ce3 thru 0x028fab (0x62c9 bytes) 000536 1789674907.378067 write 0x025b02 thru 0x02c1c7 (0x66c6 bytes) 000537 1789674907.390469 write 0x021ce4 thru 0x023f15 (0x2232 bytes) 000538 1789674907.395656 MIRROR_RESYNC 7 mirrors 000540 1789674910.219258 punch from 0x02e08a to 0x0358b5 (0x782c bytes) 000541 1789674910.311552 trunc from 0x0358b6 to 0x02c24e (0x9667 bytes) 000542 1789674910.356610 MIRROR_RESYNC 7 mirrors 000544 1789674912.100026 trunc from 0x02c24f to 0x01bdef (0x1045f bytes) 000547 1789674912.181794 punch from 0x00a8cc to 0x018b78 (0xe2ad bytes) 000550 1789674912.195593 punch from 0x008d5a to 0x00d8cb (0x4b72 bytes) 000551 1789674912.205806 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1bdf0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bdf0): Mirror 4: 0xb8149a1c (stale) Mirror 5: 0xb8149a1c (stale) Mirror 6: 0xb8149a1c (stale) Mirror 7: 0xb8149a1c (stale) Mirror 8: 0xb8149a1c (stale) Mirror 9: 0xb8149a1c (stale) CRC-32 checksum value for chunk [0, 0x1bdf0): Mirror 1: 0xb8149a1c Mirror 10: 0xb8149a1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bdf0: skipped 000552 1789674914.076507 mapwrite 0x01ab3b thru 0x024142 (0x9608 bytes) 000553 1789674914.523049 mapread 0x01a6e7 thru 0x021065 (0x697f bytes) 000554 1789674914.525074 punch from 0x017350 to 0x0202fa (0x8fab bytes) 000555 1789674914.534137 read 0x00f6c6 thru 0x0150e2 (0x5a1d bytes) 000556 1789674914.546186 read 0x018112 thru 0x023037 (0xaf26 bytes) 000557 1789674914.612036 read 0x007d9a thru 0x0145da (0xc841 bytes) 000560 1789674914.620108 mapwrite 0x020e0d thru 0x024fb4 (0x41a8 bytes) 000561 1789674914.749442 mapread 0x01f6dc thru 0x024fb4 (0x58d9 bytes) 000563 1789674914.757832 mapread 0x02463d thru 0x024fb4 (0x978 bytes) 000564 1789674914.762881 trunc from 0x024fb5 to 0x03d18e (0x181da bytes) 000566 1789674914.801333 punch from 0x022586 to 0x0246b4 (0x212f bytes) 000567 1789674914.813318 punch from 0x00f5de to 0x0152c2 (0x5ce5 bytes) 000568 1789674914.821161 punch from 0x0332f0 to 0x03d18e (0x9e9f bytes) 000569 1789674914.833724 trunc from 0x03d18f to 0x0083aa (0x34de4 bytes) 000571 1789674914.860082 read 0x0013c1 thru 0x0083aa (0x6fea bytes) 000572 1789674914.862593 write 0x02c68d thru 0x02dfbc (0x1930 bytes) 000573 1789674914.864687 punch from 0x00a93e to 0x00e189 (0x384c bytes) 000575 1789674914.910828 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dfbd): Mirror 4: 0x868b9a8b (stale) Mirror 5: 0x868b9a8b (stale) Mirror 6: 0x868b9a8b (stale) Mirror 7: 0x868b9a8b (stale) Mirror 8: 0x868b9a8b (stale) Mirror 9: 0x868b9a8b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dfbd): Mirror 4: 0x868b9a8b (stale) Mirror 5: 0x868b9a8b (stale) Mirror 6: 0x868b9a8b (stale) Mirror 7: 0x868b9a8b (stale) Mirror 8: 0x868b9a8b (stale) Mirror 9: 0x868b9a8b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dfbd: skipped 000576 1789674915.675271 write 0x02937c thru 0x03199c (0x8621 bytes) 000577 1789674915.692457 read 0x0308e6 thru 0x03199c (0x10b7 bytes) 000578 1789674915.693069 mapwrite 0x02308e thru 0x027742 (0x46b5 bytes) 000579 1789674915.828092 mapwrite 0x03a3e4 thru 0x03ffff (0x5c1c bytes) 000581 1789674916.101716 mapwrite 0x01ac8c thru 0x027356 (0xc6cb bytes) 000582 1789674916.295273 write 0x033623 thru 0x034595 (0xf73 bytes) truncating to largest ever: 0x3def5 000583 1789674916.297235 trunc from 0x040000 to 0x03def4 (0x210b bytes) 000584 1789674916.371715 MIRROR_RESYNC 7 mirrors 000585 1789674919.773835 mapread 0x03c3af thru 0x03def4 (0x1b46 bytes) 000586 1789674919.792352 mapread 0x0117ea thru 0x0143e7 (0x2bfe bytes) 000588 1789674919.820230 mapwrite 0x02433a thru 0x025841 (0x1508 bytes) 000589 1789674920.264450 read 0x00033d thru 0x00b770 (0xb434 bytes) 000591 1789674920.288172 trunc from 0x03def5 to 0x03ba1c (0x24d8 bytes) 000592 1789674920.381691 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ba1d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba1d): Mirror 4: 0xd3dc433c (stale) Mirror 5: 0xd3dc433c (stale) Mirror 6: 0xd3dc433c (stale) Mirror 7: 0xd3dc433c (stale) Mirror 8: 0xd3dc433c (stale) Mirror 9: 0xd3dc433c (stale) CRC-32 checksum value for chunk [0, 0x3ba1d): Mirror 1: 0xa81a8727 Mirror 10: 0xa81a8727 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba1d: skipped 000593 1789674922.407489 punch from 0x01e71c to 0x02e21b (0xfb00 bytes) 000594 1789674922.560774 write 0x029c9b thru 0x02d325 (0x368b bytes) 000596 1789674922.580946 MIRROR_RESYNC 8 mirrors 000597 1789674928.024323 trunc from 0x03ba1d to 0x02df10 (0xdb0c bytes) 000599 1789674928.136301 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2df11): Mirror 4: 0xe805b9f (stale) Mirror 5: 0xe805b9f (stale) Mirror 6: 0xe805b9f (stale) Mirror 7: 0xe805b9f (stale) Mirror 8: 0xe805b9f (stale) Mirror 9: 0xe805b9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2df11): Mirror 4: 0xe805b9f (stale) Mirror 5: 0xe805b9f (stale) Mirror 6: 0xe805b9f (stale) Mirror 7: 0xe805b9f (stale) Mirror 8: 0xe805b9f (stale) Mirror 9: 0xe805b9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2df11: skipped 000602 1789674928.876561 read 0x01f7a1 thru 0x02df10 (0xe770 bytes) 000603 1789674928.889947 write 0x009e43 thru 0x00f207 (0x53c5 bytes) 000604 1789674928.900715 trunc from 0x02df11 to 0x0083c4 (0x25b4c bytes) 000606 1789674928.920690 trunc from 0x0083c5 to 0x001d5a (0x666a bytes) 000607 1789674928.944972 mapwrite 0x037699 thru 0x03ffff (0x8967 bytes) 000608 1789674929.043661 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x51ed8ed1 (stale) Mirror 6: 0x51ed8ed1 (stale) Mirror 7: 0x51ed8ed1 (stale) Mirror 8: 0x51ed8ed1 (stale) Mirror 9: 0x51ed8ed1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x51ed8ed1 (stale) Mirror 6: 0x51ed8ed1 (stale) Mirror 7: 0x51ed8ed1 (stale) Mirror 8: 0x51ed8ed1 (stale) Mirror 9: 0x51ed8ed1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000609 1789674929.669863 punch from 0x02a926 to 0x02c6bb (0x1d96 bytes) 000610 1789674929.678241 MIRROR_RESYNC 6 mirrors 000611 1789674931.485256 mapread 0x01c168 thru 0x025551 (0x93ea bytes) 000614 1789674931.520196 MIRROR_EXTEND to 7 mirrors 000615 1789674932.497846 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000616 1789674932.983162 write 0x0291da thru 0x02d746 (0x456d bytes) 000617 1789674933.228549 punch from 0x031dcd to 0x03ffff (0xe233 bytes) 000618 1789674933.255449 trunc from 0x040000 to 0x0316a9 (0xe956 bytes) 000620 1789674933.290125 write 0x021cdf thru 0x02759c (0x58be bytes) 000621 1789674933.316462 mapwrite 0x01bc08 thru 0x01fe6f (0x4268 bytes) 000622 1789674933.495227 trunc from 0x0316aa to 0x0095f3 (0x280b6 bytes) 000624 1789674933.516226 mapread 0x003428 thru 0x0095f3 (0x61cc bytes) 000625 1789674933.536626 MIRROR_RESYNC 6 mirrors 000626 1789674934.913350 mapread 0x007a1c thru 0x0095f3 (0x1bd8 bytes) 000627 1789674934.942993 punch from 0x00520d to 0x007c48 (0x2a3c bytes) 000628 1789674935.055719 read 0x0029a2 thru 0x0095f3 (0x6c52 bytes) 000629 1789674935.069780 read 0x000d75 thru 0x004c6e (0x3efa bytes) 000630 1789674935.077541 mapread 0x008fac thru 0x0095f3 (0x648 bytes) 000632 1789674935.079879 write 0x00d5a0 thru 0x01b74a (0xe1ab bytes) 000634 1789674935.094022 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1b74b): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b74b): Mirror 5: 0xca9b4626 (stale) Mirror 6: 0xca9b4626 (stale) Mirror 7: 0xca9b4626 (stale) Mirror 8: 0xca9b4626 (stale) Mirror 9: 0xca9b4626 (stale) CRC-32 checksum value for chunk [0, 0x1b74b): Mirror 1: 0xeda6b763 Mirror 10: 0xeda6b763 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b74b: skipped 000636 1789674935.962273 mapread 0x0037e2 thru 0x00d1b1 (0x99d0 bytes) 000637 1789674935.989061 write 0x0256f6 thru 0x02ccea (0x75f5 bytes) 000639 1789674936.070571 write 0x03301c thru 0x038e1b (0x5e00 bytes) 000640 1789674936.071834 trunc from 0x038e1c to 0x03a640 (0x1825 bytes) 000641 1789674936.113175 MIRROR_RESYNC 7 mirrors 000642 1789674937.196615 MIRROR_RESYNC 7 mirrors 000643 1789674937.614006 punch from 0x003673 to 0x005bcc (0x255a bytes) 000645 1789674937.717905 write 0x03623e thru 0x03c802 (0x65c5 bytes) 000646 1789674937.725128 mapwrite 0x01e7a5 thru 0x0291ad (0xaa09 bytes) 000647 1789674937.932161 MIRROR_RESYNC 7 mirrors 000648 1789674939.382339 punch from 0x009787 to 0x00c83e (0x30b8 bytes) 000649 1789674939.441672 punch from 0x0192cc to 0x022d28 (0x9a5d bytes) 000651 1789674939.473910 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c803): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c803): Mirror 5: 0xa7119cb8 (stale) Mirror 6: 0xa7119cb8 (stale) Mirror 7: 0xa7119cb8 (stale) Mirror 8: 0xa7119cb8 (stale) Mirror 9: 0xa7119cb8 (stale) Mirror 10: 0xa7119cb8 (stale) CRC-32 checksum value for chunk [0, 0x3c803): Mirror 1: 0x61611a33 Mirror 11: 0x61611a33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c803: skipped 000652 1789674941.096031 read 0x03986c thru 0x03c802 (0x2f97 bytes) 000654 1789674941.117042 trunc from 0x03c803 to 0x00e5a3 (0x2e25f bytes) 000656 1789674941.220309 write 0x00217f thru 0x00c602 (0xa484 bytes) 000658 1789674941.235350 write 0x02e9db thru 0x035526 (0x6b4c bytes) 000661 1789674941.247138 MIRROR_RESYNC 8 mirrors 000662 1789674943.784892 write 0x0046bf thru 0x00b2c5 (0x6c07 bytes) 000663 1789674943.871052 write 0x00a3cc thru 0x019319 (0xef4e bytes) 000664 1789674943.883566 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x35527): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35527): Mirror 5: 0x8d680950 (stale) Mirror 6: 0x8d680950 (stale) Mirror 7: 0x8d680950 (stale) Mirror 8: 0x8d680950 (stale) Mirror 9: 0x8d680950 (stale) Mirror 10: 0x8d680950 (stale) Mirror 11: 0x8d680950 (stale) CRC-32 checksum value for chunk [0, 0x35527): Mirror 1: 0x91961a28 Mirror 12: 0x91961a28 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35527: skipped 000665 1789674945.429726 mapwrite 0x0051fc thru 0x012d7f (0xdb84 bytes) 000670 1789674945.672463 mapread 0x030758 thru 0x035526 (0x4dcf bytes) 000671 1789674945.690155 mapread 0x01b583 thru 0x0250b4 (0x9b32 bytes) 000674 1789674945.707047 read 0x0341e1 thru 0x035526 (0x1346 bytes) 000675 1789674945.707695 write 0x005cc6 thru 0x012f4a (0xd285 bytes) 000676 1789674945.709759 mapwrite 0x00ec7e thru 0x01eb4a (0xfecd bytes) 000677 1789674945.758908 read 0x01b99f thru 0x027037 (0xb699 bytes) 000679 1789674945.760196 mapwrite 0x00e3cc thru 0x0154d0 (0x7105 bytes) 000680 1789674945.807060 mapwrite 0x018f38 thru 0x01d79a (0x4863 bytes) 000682 1789674945.855476 mapwrite 0x0365a4 thru 0x039819 (0x3276 bytes) 000684 1789674945.928561 write 0x03c249 thru 0x03ffff (0x3db7 bytes) 000686 1789674945.931032 trunc from 0x040000 to 0x027c04 (0x183fb bytes) 000687 1789674945.951706 mapwrite 0x002a58 thru 0x010d85 (0xe32e bytes) 000688 1789674946.001144 read 0x01b8dc thru 0x01f3a2 (0x3ac7 bytes) 000689 1789674946.001862 write 0x01b3d7 thru 0x02014e (0x4d78 bytes) 000693 1789674946.006318 trunc from 0x027c05 to 0x00e5ad (0x19657 bytes) 000695 1789674946.024437 mapwrite 0x005dd0 thru 0x00a8ea (0x4b1b bytes) 000696 1789674946.077537 mapread 0x00a923 thru 0x00e5ad (0x3c8b bytes) 000697 1789674946.081036 punch from 0x002a56 to 0x00787a (0x4e25 bytes) 000698 1789674946.088902 write 0x0249e2 thru 0x031038 (0xc657 bytes) 000699 1789674946.091193 punch from 0x00113c to 0x00e618 (0xd4dd bytes) 000700 1789674946.107635 punch from 0x00edd2 to 0x01a193 (0xb3c2 bytes) 000703 1789674946.119055 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x31039): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31039): Mirror 5: 0xe02fda39 (stale) Mirror 6: 0xe02fda39 (stale) Mirror 7: 0xe02fda39 (stale) Mirror 8: 0xe02fda39 (stale) Mirror 9: 0xe02fda39 (stale) Mirror 10: 0xe02fda39 (stale) Mirror 11: 0xe02fda39 (stale) Mirror 12: 0x41435c89 (stale) CRC-32 checksum value for chunk [0, 0x31039): Mirror 1: 0x4497e319 Mirror 13: 0x4497e319 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31039: skipped 000704 1789674947.316048 write 0x003be8 thru 0x01373d (0xfb56 bytes) 000705 1789674947.447278 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x31039): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31039): Mirror 5: 0xe02fda39 (stale) Mirror 6: 0xe02fda39 (stale) Mirror 7: 0xe02fda39 (stale) Mirror 8: 0xe02fda39 (stale) Mirror 9: 0xe02fda39 (stale) Mirror 10: 0xe02fda39 (stale) Mirror 11: 0xe02fda39 (stale) Mirror 12: 0x41435c89 (stale) Mirror 13: 0x4497e319 (stale) CRC-32 checksum value for chunk [0, 0x31039): Mirror 1: 0x2e20b736 Mirror 14: 0x2e20b736 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31039: skipped 000706 1789674949.297069 punch from 0x00a3ce to 0x010c0d (0x6840 bytes) 000707 1789674949.416817 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x31039): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31039): Mirror 5: 0xe02fda39 (stale) Mirror 6: 0xe02fda39 (stale) Mirror 7: 0xe02fda39 (stale) Mirror 8: 0xe02fda39 (stale) Mirror 9: 0xe02fda39 (stale) Mirror 10: 0xe02fda39 (stale) Mirror 11: 0xe02fda39 (stale) Mirror 12: 0x41435c89 (stale) Mirror 13: 0x4497e319 (stale) Mirror 14: 0x2e20b736 (stale) CRC-32 checksum value for chunk [0, 0x31039): Mirror 1: 0x8c4d640 Mirror 15: 0x8c4d640 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31039: skipped 000708 1789674950.629028 write 0x01ca1f thru 0x02c129 (0xf70b bytes) 000709 1789674950.714749 write 0x015d20 thru 0x0256ed (0xf9ce bytes) 000711 1789674950.719682 write 0x030663 thru 0x03ad83 (0xa721 bytes) 000713 1789674950.725046 trunc from 0x03ad84 to 0x038652 (0x2731 bytes) 000714 1789674950.737738 trunc from 0x038653 to 0x03353a (0x5118 bytes) 000715 1789674950.752056 read 0x013cd6 thru 0x02101b (0xd346 bytes) 000716 1789674950.756156 trunc from 0x03353b to 0x00bca1 (0x27899 bytes) 000720 1789674950.772027 trunc from 0x00bca2 to 0x00eefe (0x325d bytes) 000721 1789674950.788086 punch from 0x00ea34 to 0x00eefe (0x4cb bytes) 000723 1789674950.793272 read 0x009a11 thru 0x009a93 (0x83 bytes) 000725 1789674950.810318 mapwrite 0x02cfdf thru 0x035f88 (0x8faa bytes) 000726 1789674950.965140 punch from 0x00eebd to 0x019e4a (0xaf8e bytes) 000727 1789674950.978232 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35f89): Mirror 5: 0x28d19968 (stale) Mirror 6: 0x28d19968 (stale) Mirror 7: 0x28d19968 (stale) Mirror 8: 0x28d19968 (stale) Mirror 9: 0x28d19968 (stale) Mirror 10: 0x28d19968 (stale) Mirror 11: 0x28d19968 (stale) Mirror 12: 0x8fc4ca8f (stale) Mirror 13: 0x45ac0ee9 (stale) Mirror 14: 0xfd5b4a13 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f89): Mirror 5: 0x28d19968 (stale) Mirror 6: 0x28d19968 (stale) Mirror 7: 0x28d19968 (stale) Mirror 8: 0x28d19968 (stale) Mirror 9: 0x28d19968 (stale) Mirror 10: 0x28d19968 (stale) Mirror 11: 0x28d19968 (stale) Mirror 12: 0x8fc4ca8f (stale) Mirror 13: 0x45ac0ee9 (stale) Mirror 14: 0xfd5b4a13 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f89: skipped 000728 1789674951.741307 punch from 0x00526d to 0x00dced (0x8a81 bytes) 000729 1789674951.749273 read 0x0278f2 thru 0x033155 (0xb864 bytes) 000732 1789674951.772970 mapread 0x0138f0 thru 0x01794e (0x405f bytes) 000733 1789674951.808258 read 0x0255e3 thru 0x02be00 (0x681e bytes) 000734 1789674951.843950 write 0x00be9e thru 0x011246 (0x53a9 bytes) 000735 1789674951.849942 trunc from 0x035f89 to 0x038404 (0x247c bytes) 000737 1789674951.881859 mapread 0x0203f3 thru 0x02e69d (0xe2ab bytes) 000738 1789674951.884811 mapread 0x027636 thru 0x0362f2 (0xecbd bytes) 000739 1789674951.892992 punch from 0x028c12 to 0x030937 (0x7d26 bytes) 000742 1789674951.917003 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38405): Mirror 5: 0x3ccfc1b6 (stale) Mirror 6: 0x3ccfc1b6 (stale) Mirror 7: 0x3ccfc1b6 (stale) Mirror 8: 0x3ccfc1b6 (stale) Mirror 9: 0x3ccfc1b6 (stale) Mirror 10: 0x3ccfc1b6 (stale) Mirror 12: 0x6448edc4 (stale) Mirror 13: 0x81df054c (stale) Mirror 14: 0x1d8bcb9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38405): Mirror 5: 0x3ccfc1b6 (stale) Mirror 6: 0x3ccfc1b6 (stale) Mirror 7: 0x3ccfc1b6 (stale) Mirror 8: 0x3ccfc1b6 (stale) Mirror 9: 0x3ccfc1b6 (stale) Mirror 10: 0x3ccfc1b6 (stale) Mirror 12: 0x6448edc4 (stale) Mirror 13: 0x81df054c (stale) Mirror 14: 0x1d8bcb9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38405: skipped 000744 1789674952.514461 mapread 0x02092a thru 0x02c5dd (0xbcb4 bytes) 000746 1789674952.547318 trunc from 0x038405 to 0x019ed4 (0x1e530 bytes) 000747 1789674952.570037 mapread 0x001ca8 thru 0x00f086 (0xd3df bytes) 000748 1789674952.580943 punch from 0x0049ad to 0x00dc52 (0x92a6 bytes) 000750 1789674952.586306 read 0x00d85f thru 0x016f9a (0x973c bytes) 000751 1789674952.599520 mapwrite 0x00bccd thru 0x01b524 (0xf858 bytes) 000753 1789674952.696817 mapwrite 0x0119a3 thru 0x015d46 (0x43a4 bytes) 000755 1789674952.755293 read 0x011880 thru 0x01b524 (0x9ca5 bytes) 000756 1789674952.756171 write 0x03b65e thru 0x03ffff (0x49a2 bytes) 000757 1789674952.758002 punch from 0x00f05b to 0x01cd0f (0xdcb5 bytes) 000759 1789674952.771248 read 0x01a4a4 thru 0x01d9e0 (0x353d bytes) 000762 1789674952.785040 mapwrite 0x0255f9 thru 0x02b126 (0x5b2e bytes) 000763 1789674952.831174 punch from 0x03f624 to 0x03ffff (0x9dc bytes) 000764 1789674952.841901 trunc from 0x040000 to 0x00521b (0x3ade4 bytes) 000765 1789674952.859806 punch from 0x0034ad to 0x00521b (0x1d6f bytes) 000766 1789674952.865490 punch from 0x000a8e to 0x00521b (0x478e bytes) 000767 1789674952.870984 mapwrite 0x0276c1 thru 0x029155 (0x1a95 bytes) 000768 1789674952.971911 trunc from 0x029156 to 0x01d607 (0xbb4e bytes) 000769 1789674952.998948 trunc from 0x01d608 to 0x00c738 (0x10ecf bytes) 000771 1789674953.021759 mapwrite 0x00f9ca thru 0x01cfa3 (0xd5da bytes) 000772 1789674953.147036 mapread 0x009caf thru 0x00e65b (0x49ad bytes) 000773 1789674953.187335 write 0x013c0c thru 0x01d10a (0x94ff bytes) 000774 1789674953.189214 trunc from 0x01d10b to 0x029912 (0xc808 bytes) 000775 1789674953.205650 mapread 0x021ca8 thru 0x02224e (0x5a7 bytes) 000776 1789674953.219909 mapwrite 0x026c8d thru 0x0282fc (0x1670 bytes) 000777 1789674953.294710 trunc from 0x029913 to 0x034c3f (0xb32d bytes) 000778 1789674953.314960 punch from 0x00ad72 to 0x01a423 (0xf6b2 bytes) 000779 1789674953.320847 mapwrite 0x01e1f2 thru 0x01f3f7 (0x1206 bytes) 000780 1789674953.375532 punch from 0x0160d4 to 0x023ccd (0xdbfa bytes) 000781 1789674953.381941 mapread 0x021ae6 thru 0x022f5c (0x1477 bytes) 000782 1789674953.392201 punch from 0x030743 to 0x031d29 (0x15e7 bytes) 000783 1789674953.399849 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x34c40): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34c40): Mirror 5: 0xff897b33 (stale) Mirror 6: 0xff897b33 (stale) Mirror 7: 0xff897b33 (stale) Mirror 8: 0xff897b33 (stale) Mirror 9: 0xff897b33 (stale) Mirror 10: 0xff897b33 (stale) Mirror 12: 0xcfe001db (stale) Mirror 13: 0x3cd3c48c (stale) Mirror 14: 0xdaf41cbf (stale) CRC-32 checksum value for chunk [0, 0x34c40): Mirror 1: 0x563db2a0 Mirror 15: 0x563db2a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34c40: skipped 000786 1789674955.136753 trunc from 0x034c40 to 0x008ce7 (0x2bf58 bytes) 000788 1789674955.281221 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x8ce8): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8ce8): Mirror 5: 0xddb5db29 (stale) Mirror 6: 0xddb5db29 (stale) Mirror 7: 0xddb5db29 (stale) Mirror 8: 0xddb5db29 (stale) Mirror 9: 0xddb5db29 (stale) Mirror 10: 0xddb5db29 (stale) Mirror 12: 0x7c598349 (stale) Mirror 13: 0xe3886081 (stale) Mirror 14: 0x97ba340f (stale) Mirror 15: 0xe3886081 (stale) CRC-32 checksum value for chunk [0, 0x8ce8): Mirror 1: 0xe3886081 Mirror 16: 0xe3886081 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8ce8: skipped 000789 1789674956.582289 mapread 0x0088d1 thru 0x008ce7 (0x417 bytes) 000791 1789674956.671735 MIRROR_RESYNC 12 mirrors 000792 1789674959.936673 write 0x0047eb thru 0x013fb0 (0xf7c6 bytes) 000793 1789674960.053865 trunc from 0x013fb1 to 0x035ca0 (0x21cf0 bytes) 000795 1789674960.109213 trunc from 0x035ca1 to 0x01bd3d (0x19f63 bytes) 000797 1789674960.175880 write 0x020559 thru 0x026934 (0x63dc bytes) 000799 1789674960.180853 punch from 0x00180b to 0x00c8e1 (0xb0d7 bytes) 000800 1789674960.239774 mapread 0x00643a thru 0x010784 (0xa34b bytes) 000801 1789674960.271618 trunc from 0x026935 to 0x01cf9b (0x9999 bytes) 000803 1789674960.290420 mapwrite 0x0087b8 thru 0x00926d (0xab6 bytes) 000804 1789674960.344875 write 0x0363c2 thru 0x03ecd6 (0x8915 bytes) 000805 1789674960.352882 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3ecd7): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecd7): Mirror 5: 0xce90941a (stale) Mirror 6: 0xce90941a (stale) Mirror 7: 0xce90941a (stale) Mirror 8: 0xce90941a (stale) Mirror 9: 0xce90941a (stale) Mirror 10: 0xce90941a (stale) Mirror 12: 0xce90941a (stale) Mirror 13: 0xce90941a (stale) Mirror 14: 0xce90941a (stale) Mirror 15: 0xce90941a (stale) Mirror 16: 0xce90941a (stale) CRC-32 checksum value for chunk [0, 0x3ecd7): Mirror 1: 0x2142a6db Mirror 17: 0x2142a6db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecd7: skipped 000806 1789674961.754826 read 0x0056b6 thru 0x00bd8a (0x66d5 bytes) 000807 1789674961.790828 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ecd7): Mirror 5: 0xce90941a (stale) Mirror 6: 0xce90941a (stale) Mirror 7: 0xce90941a (stale) Mirror 8: 0xce90941a (stale) Mirror 9: 0xce90941a (stale) Mirror 10: 0xce90941a (stale) Mirror 12: 0xce90941a (stale) Mirror 13: 0xce90941a (stale) Mirror 15: 0xce90941a (stale) Mirror 16: 0xce90941a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecd7): Mirror 5: 0xce90941a (stale) Mirror 6: 0xce90941a (stale) Mirror 7: 0xce90941a (stale) Mirror 8: 0xce90941a (stale) Mirror 9: 0xce90941a (stale) Mirror 10: 0xce90941a (stale) Mirror 12: 0xce90941a (stale) Mirror 13: 0xce90941a (stale) Mirror 15: 0xce90941a (stale) Mirror 16: 0xce90941a (stale) CRC-32 checksum value for chunk [0, 0x3ecd7): Mirror 1: 0x2142a6db Mirror 17: 0x2142a6db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecd7: skipped 000808 1789674963.347322 mapread 0x032c03 thru 0x03a880 (0x7c7e bytes) 000813 1789674963.548236 trunc from 0x03ecd7 to 0x03b28e (0x3a48 bytes) 000814 1789674963.790883 MIRROR_RESYNC 12 mirrors 000815 1789674966.823196 read 0x01cf3d thru 0x02a5d2 (0xd696 bytes) 000816 1789674966.838023 MIRROR_EXTEND to 13 mirrors 000817 1789674967.716354 mapwrite 0x005a3d thru 0x00804b (0x260f bytes) 000818 1789674967.950405 write 0x009733 thru 0x015ff5 (0xc8c3 bytes) 000819 1789674967.967821 trunc from 0x03b28f to 0x01ebfb (0x1c693 bytes) 000820 1789674967.997299 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ebfc): Mirror 5: 0xed3aa969 (stale) Mirror 6: 0xed3aa969 (stale) Mirror 7: 0xed3aa969 (stale) Mirror 8: 0xed3aa969 (stale) Mirror 9: 0xed3aa969 (stale) Mirror 10: 0xed3aa969 (stale) Mirror 13: 0xed3aa969 (stale) Mirror 15: 0xed3aa969 (stale) Mirror 16: 0xed3aa969 (stale) Mirror 17: 0xed3aa969 (stale) Mirror 18: 0xed3aa969 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ebfc): Mirror 5: 0xed3aa969 (stale) Mirror 6: 0xed3aa969 (stale) Mirror 7: 0xed3aa969 (stale) Mirror 8: 0xed3aa969 (stale) Mirror 9: 0xed3aa969 (stale) Mirror 10: 0xed3aa969 (stale) Mirror 13: 0xed3aa969 (stale) Mirror 15: 0xed3aa969 (stale) Mirror 16: 0xed3aa969 (stale) Mirror 17: 0xed3aa969 (stale) Mirror 18: 0xed3aa969 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ebfc: skipped 000821 1789674968.845454 mapread 0x00997e thru 0x01417f (0xa802 bytes) 000822 1789674968.888171 mapwrite 0x00816d thru 0x0081bd (0x51 bytes) 000823 1789674968.935845 mapwrite 0x028427 thru 0x03781c (0xf3f6 bytes) 000824 1789674969.036922 trunc from 0x03781d to 0x01227d (0x2559f bytes) 000825 1789674969.057815 mapwrite 0x017634 thru 0x025db3 (0xe780 bytes) 000828 1789674969.157291 mapread 0x013a84 thru 0x01d493 (0x9a10 bytes) 000830 1789674969.163614 trunc from 0x025db4 to 0x0326e2 (0xc92f bytes) 000832 1789674969.186716 punch from 0x009364 to 0x017fb4 (0xec51 bytes) 000833 1789674969.192888 punch from 0x014d92 to 0x015dea (0x1059 bytes) 000835 1789674969.202696 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x326e3): Mirror 5: 0x5940c9e3 (stale) Mirror 6: 0x5940c9e3 (stale) Mirror 7: 0x5940c9e3 (stale) Mirror 8: 0x5940c9e3 (stale) Mirror 9: 0x5940c9e3 (stale) Mirror 10: 0x5940c9e3 (stale) Mirror 13: 0x5940c9e3 (stale) Mirror 15: 0x5940c9e3 (stale) Mirror 16: 0x5940c9e3 (stale) Mirror 17: 0x5940c9e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x326e3): Mirror 5: 0x5940c9e3 (stale) Mirror 6: 0x5940c9e3 (stale) Mirror 7: 0x5940c9e3 (stale) Mirror 8: 0x5940c9e3 (stale) Mirror 9: 0x5940c9e3 (stale) Mirror 10: 0x5940c9e3 (stale) Mirror 13: 0x5940c9e3 (stale) Mirror 15: 0x5940c9e3 (stale) Mirror 16: 0x5940c9e3 (stale) Mirror 17: 0x5940c9e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x326e3: skipped 000836 1789674969.964611 trunc from 0x0326e3 to 0x00ed0c (0x239d6 bytes) 000837 1789674969.978181 punch from 0x0050c3 to 0x00c55e (0x749c bytes) 000838 1789674969.982864 mapwrite 0x018c28 thru 0x02441a (0xb7f3 bytes) 000839 1789674970.113114 mapread 0x018bf5 thru 0x01ad21 (0x212d bytes) 000840 1789674970.115612 mapread 0x00b8e1 thru 0x014513 (0x8c33 bytes) 000841 1789674970.138116 trunc from 0x02441b to 0x012553 (0x11ec7 bytes) 000842 1789674970.161475 read 0x00114f thru 0x00ee7a (0xdd2c bytes) 000843 1789674970.168848 mapread 0x002cc3 thru 0x005327 (0x2665 bytes) 000846 1789674970.170711 mapwrite 0x033b16 thru 0x03ffff (0xc4ea bytes) 000847 1789674970.295596 write 0x01086b thru 0x011047 (0x7dd bytes) 000848 1789674970.297808 punch from 0x008805 to 0x014e3c (0xc638 bytes) 000849 1789674970.315475 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xce977eb4 (stale) Mirror 7: 0xce977eb4 (stale) Mirror 8: 0xce977eb4 (stale) Mirror 9: 0xce977eb4 (stale) Mirror 10: 0xce977eb4 (stale) Mirror 13: 0xce977eb4 (stale) Mirror 15: 0xce977eb4 (stale) Mirror 16: 0xce977eb4 (stale) Mirror 17: 0xce977eb4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xce977eb4 (stale) Mirror 7: 0xce977eb4 (stale) Mirror 8: 0xce977eb4 (stale) Mirror 9: 0xce977eb4 (stale) Mirror 10: 0xce977eb4 (stale) Mirror 13: 0xce977eb4 (stale) Mirror 15: 0xce977eb4 (stale) Mirror 16: 0xce977eb4 (stale) Mirror 17: 0xce977eb4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000851 1789674971.228538 mapread 0x00d69f thru 0x01aac5 (0xd427 bytes) 000852 1789674971.263853 mapread 0x01073c thru 0x017803 (0x70c8 bytes) 000855 1789674971.268144 write 0x02e338 thru 0x0306ff (0x23c8 bytes) 000856 1789674971.288555 mapwrite 0x000393 thru 0x00e10a (0xdd78 bytes) 000857 1789674971.443688 write 0x03b193 thru 0x03ffff (0x4e6d bytes) 000858 1789674971.451614 mapwrite 0x03bda5 thru 0x03ffff (0x425b bytes) 000859 1789674971.534748 read 0x006f46 thru 0x00732f (0x3ea bytes) 000860 1789674971.536857 mapwrite 0x01feaf thru 0x0228bf (0x2a11 bytes) 000861 1789674971.689923 punch from 0x013ecc to 0x01f0e1 (0xb216 bytes) 000862 1789674971.702708 mapwrite 0x0267b9 thru 0x02fafa (0x9342 bytes) 000864 1789674971.819904 mapwrite 0x03299d thru 0x03cce5 (0xa349 bytes) 000865 1789674971.911792 punch from 0x030a60 to 0x036b31 (0x60d2 bytes) 000868 1789674971.923713 mapwrite 0x012df7 thru 0x01d805 (0xaa0f bytes) 000869 1789674972.045848 mapwrite 0x025930 thru 0x02cfa3 (0x7674 bytes) 000870 1789674972.105312 trunc from 0x040000 to 0x01dc00 (0x223ff bytes) 000871 1789674972.143643 mapwrite 0x00a029 thru 0x00ee94 (0x4e6c bytes) 000873 1789674972.204632 write 0x024a17 thru 0x029d16 (0x5300 bytes) 000874 1789674972.207861 mapwrite 0x003325 thru 0x011aa0 (0xe77c bytes) 000875 1789674972.274790 punch from 0x020ca7 to 0x02533e (0x4698 bytes) 000876 1789674972.280539 read 0x01d879 thru 0x029d16 (0xc49e bytes) 000878 1789674972.293262 punch from 0x024827 to 0x0258ed (0x10c7 bytes) 000880 1789674972.299335 mapread 0x0172af thru 0x026af8 (0xf84a bytes) 000883 1789674972.313750 trunc from 0x029d17 to 0x005425 (0x248f1 bytes) 000884 1789674972.331664 punch from 0x00022f to 0x005425 (0x51f7 bytes) 000888 1789674972.341012 punch from 0x001d27 to 0x005425 (0x36ff bytes) 000892 1789674972.348046 trunc from 0x005426 to 0x029bb0 (0x2478b bytes) 000893 1789674972.364811 trunc from 0x029bb1 to 0x03a5c0 (0x10a10 bytes) 000894 1789674972.384512 punch from 0x0067d2 to 0x00eed4 (0x8703 bytes) 000896 1789674972.390511 punch from 0x008750 to 0x012854 (0xa105 bytes) 000897 1789674972.395767 punch from 0x036a4c to 0x03a5c0 (0x3b75 bytes) 000898 1789674972.401731 read 0x01f959 thru 0x024cdb (0x5383 bytes) 000899 1789674972.418530 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3a5c1): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5c1): Mirror 6: 0xe2be6db6 (stale) Mirror 7: 0xe2be6db6 (stale) Mirror 8: 0xe2be6db6 (stale) Mirror 9: 0xe2be6db6 (stale) Mirror 10: 0xe2be6db6 (stale) Mirror 13: 0xe2be6db6 (stale) Mirror 15: 0xe2be6db6 (stale) Mirror 16: 0xe2be6db6 (stale) Mirror 17: 0xe2be6db6 (stale) CRC-32 checksum value for chunk [0, 0x3a5c1): Mirror 1: 0xced29357 Mirror 18: 0xced29357 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5c1: skipped 000900 1789674974.830995 mapread 0x009380 thru 0x01461b (0xb29c bytes) 000901 1789674974.881129 mapwrite 0x0231cb thru 0x02ae22 (0x7c58 bytes) 000902 1789674975.055721 mapread 0x039ba4 thru 0x03a5c0 (0xa1d bytes) 000903 1789674975.065879 mapread 0x012086 thru 0x01f10a (0xd085 bytes) 000904 1789674975.085968 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a5c1): Mirror 6: 0xe2be6db6 (stale) Mirror 7: 0xe2be6db6 (stale) Mirror 8: 0xe2be6db6 (stale) Mirror 9: 0xe2be6db6 (stale) Mirror 10: 0xe2be6db6 (stale) Mirror 15: 0xe2be6db6 (stale) Mirror 16: 0xe2be6db6 (stale) Mirror 17: 0xe2be6db6 (stale) Mirror 18: 0xced29357 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5c1): Mirror 6: 0xe2be6db6 (stale) Mirror 7: 0xe2be6db6 (stale) Mirror 8: 0xe2be6db6 (stale) Mirror 9: 0xe2be6db6 (stale) Mirror 10: 0xe2be6db6 (stale) Mirror 15: 0xe2be6db6 (stale) Mirror 16: 0xe2be6db6 (stale) Mirror 17: 0xe2be6db6 (stale) Mirror 18: 0xced29357 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5c1: skipped 000905 1789674975.753669 trunc from 0x03a5c1 to 0x031ed6 (0x86ea bytes) 000907 1789674975.777481 mapread 0x002e10 thru 0x00a903 (0x7af4 bytes) 000909 1789674975.804191 punch from 0x02f9bf to 0x031ed6 (0x2518 bytes) 000910 1789674975.813816 trunc from 0x031ed7 to 0x034539 (0x2663 bytes) 000911 1789674975.844495 mapwrite 0x012e7d thru 0x01b1d7 (0x835b bytes) 000912 1789674976.132018 write 0x011ea1 thru 0x017272 (0x53d2 bytes) 000913 1789674976.160584 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3453a): Mirror 7: 0x8413e624 (stale) Mirror 8: 0x8413e624 (stale) Mirror 9: 0x8413e624 (stale) Mirror 10: 0x8413e624 (stale) Mirror 15: 0x8413e624 (stale) Mirror 16: 0x8413e624 (stale) Mirror 17: 0x8413e624 (stale) Mirror 18: 0x1a3cc1a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3453a): Mirror 7: 0x8413e624 (stale) Mirror 8: 0x8413e624 (stale) Mirror 9: 0x8413e624 (stale) Mirror 10: 0x8413e624 (stale) Mirror 15: 0x8413e624 (stale) Mirror 16: 0x8413e624 (stale) Mirror 17: 0x8413e624 (stale) Mirror 18: 0x1a3cc1a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3453a: skipped 000914 1789674977.004894 mapread 0x013f55 thru 0x021485 (0xd531 bytes) 000915 1789674977.083719 mapread 0x00bfc3 thru 0x00fc17 (0x3c55 bytes) 000916 1789674977.112733 punch from 0x02bb5e to 0x02c080 (0x523 bytes) 000920 1789674977.141800 punch from 0x015343 to 0x01c0ee (0x6dac bytes) 000921 1789674977.197663 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3453a): Mirror 7: 0x8413e624 (stale) Mirror 8: 0x8413e624 (stale) Mirror 9: 0x8413e624 (stale) Mirror 15: 0x8413e624 (stale) Mirror 16: 0x8413e624 (stale) Mirror 17: 0x8413e624 (stale) Mirror 18: 0x1a3cc1a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3453a): Mirror 7: 0x8413e624 (stale) Mirror 8: 0x8413e624 (stale) Mirror 9: 0x8413e624 (stale) Mirror 15: 0x8413e624 (stale) Mirror 16: 0x8413e624 (stale) Mirror 17: 0x8413e624 (stale) Mirror 18: 0x1a3cc1a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3453a: skipped 000922 1789674978.355078 read 0x02c6ab thru 0x02f799 (0x30ef bytes) 000925 1789674978.384639 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3453a): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3453a): Mirror 7: 0x8413e624 (stale) Mirror 8: 0x8413e624 (stale) Mirror 9: 0x8413e624 (stale) Mirror 15: 0x8413e624 (stale) Mirror 16: 0x8413e624 (stale) Mirror 17: 0x8413e624 (stale) Mirror 18: 0x1a3cc1a7 (stale) CRC-32 checksum value for chunk [0, 0x3453a): Mirror 1: 0x6ec01dde Mirror 19: 0x6ec01dde lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3453a: skipped 000926 1789674980.908057 mapread 0x01d086 thru 0x028156 (0xb0d1 bytes) 000927 1789674980.983348 write 0x00eb43 thru 0x017722 (0x8be0 bytes) 000929 1789674981.143231 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3453a): Mirror 7: 0x8413e624 (stale) Mirror 9: 0x8413e624 (stale) Mirror 15: 0x8413e624 (stale) Mirror 16: 0x8413e624 (stale) Mirror 17: 0x8413e624 (stale) Mirror 18: 0x1a3cc1a7 (stale) Mirror 19: 0x6ec01dde (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3453a): Mirror 7: 0x8413e624 (stale) Mirror 9: 0x8413e624 (stale) Mirror 15: 0x8413e624 (stale) Mirror 16: 0x8413e624 (stale) Mirror 17: 0x8413e624 (stale) Mirror 18: 0x1a3cc1a7 (stale) Mirror 19: 0x6ec01dde (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3453a: skipped 000931 1789674981.692929 trunc from 0x03453a to 0x027e15 (0xc724 bytes) 000932 1789674981.709107 trunc from 0x027e16 to 0x02b375 (0x3560 bytes) 000934 1789674981.755426 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2b376): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b376): Mirror 7: 0x6491628 (stale) Mirror 9: 0x6491628 (stale) Mirror 15: 0x6491628 (stale) Mirror 16: 0x6491628 (stale) Mirror 17: 0x6491628 (stale) Mirror 18: 0x5316319d (stale) Mirror 19: 0xb72c1a21 (stale) CRC-32 checksum value for chunk [0, 0x2b376): Mirror 1: 0xe4034991 Mirror 20: 0xe4034991 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b376: skipped 000935 1789674983.402333 mapwrite 0x02f651 thru 0x03cd50 (0xd700 bytes) 000936 1789674983.896001 trunc from 0x03cd51 to 0x0383da (0x4976 bytes) 000937 1789674983.961975 MIRROR_RESYNC 9 mirrors 000938 1789674986.511748 MIRROR_EXTEND to 10 mirrors 000939 1789674987.961342 punch from 0x01fc95 to 0x027b6c (0x7ed8 bytes) 000940 1789674988.234005 mapread 0x0196f0 thru 0x01d44f (0x3d60 bytes) 000945 1789674988.276476 mapwrite 0x013206 thru 0x01de12 (0xac0d bytes) 000946 1789674988.433909 MIRROR_RESYNC 10 mirrors 000949 1789674990.158893 trunc from 0x0383db to 0x01a3ee (0x1dfec bytes) 000950 1789674990.220066 punch from 0x0143cf to 0x01a3ee (0x6020 bytes) 000951 1789674990.226119 mapwrite 0x0355f7 thru 0x03eb0f (0x9519 bytes) 000953 1789674990.351626 write 0x0223a4 thru 0x02ffed (0xdc4a bytes) 000954 1789674990.371066 write 0x01b3ec thru 0x022e31 (0x7a46 bytes) 000955 1789674990.386138 punch from 0x03e865 to 0x03eb0f (0x2ab bytes) 000956 1789674990.413729 mapread 0x01540f thru 0x01a12d (0x4d1f bytes) 000957 1789674990.435253 mapwrite 0x0114c5 thru 0x0118b6 (0x3f2 bytes) 000959 1789674990.486637 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3eb10): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb10): Mirror 7: 0xab2ad978 (stale) Mirror 9: 0xab2ad978 (stale) Mirror 15: 0xab2ad978 (stale) Mirror 16: 0xab2ad978 (stale) Mirror 17: 0xab2ad978 (stale) Mirror 18: 0xab2ad978 (stale) Mirror 19: 0xab2ad978 (stale) Mirror 20: 0xab2ad978 (stale) Mirror 21: 0xab2ad978 (stale) CRC-32 checksum value for chunk [0, 0x3eb10): Mirror 1: 0x23f45cfa Mirror 22: 0x23f45cfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb10: skipped 000960 1789674991.901008 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3eb10): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb10): Mirror 7: 0xab2ad978 (stale) Mirror 9: 0xab2ad978 (stale) Mirror 15: 0xab2ad978 (stale) Mirror 16: 0xab2ad978 (stale) Mirror 17: 0xab2ad978 (stale) Mirror 18: 0xab2ad978 (stale) Mirror 19: 0xab2ad978 (stale) Mirror 20: 0xab2ad978 (stale) Mirror 21: 0xab2ad978 (stale) CRC-32 checksum value for chunk [0, 0x3eb10): Mirror 1: 0x23f45cfa Mirror 22: 0x23f45cfa Mirror 23: 0x23f45cfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb10: skipped 000961 1789674993.279206 trunc from 0x03eb10 to 0x038bd4 (0x5f3b bytes) 000962 1789674993.413887 trunc from 0x038bd5 to 0x02ce03 (0xbdd1 bytes) 000963 1789674993.440858 MIRROR_RESYNC 12 mirrors 000964 1789674996.263653 write 0x006de3 thru 0x013be2 (0xce00 bytes) 000965 1789674996.361624 punch from 0x01aa21 to 0x02260f (0x7bef bytes) 000966 1789674996.403332 punch from 0x023e10 to 0x02ce03 (0x8ff4 bytes) 000969 1789674996.412764 read 0x02524c thru 0x02ce03 (0x7bb8 bytes) 000970 1789674996.430117 trunc from 0x02ce04 to 0x022a52 (0xa3b1 bytes) 000971 1789674996.471638 write 0x0395d1 thru 0x03ffff (0x6a2f bytes) 000973 1789674996.483247 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9bb25639 (stale) Mirror 9: 0x9bb25639 (stale) Mirror 15: 0x9bb25639 (stale) Mirror 16: 0x9bb25639 (stale) Mirror 17: 0x9bb25639 (stale) Mirror 18: 0x9bb25639 (stale) Mirror 19: 0x9bb25639 (stale) Mirror 20: 0x9bb25639 (stale) Mirror 21: 0x9bb25639 (stale) Mirror 22: 0x9bb25639 (stale) Mirror 23: 0x9bb25639 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa0a4509a Mirror 24: 0xa0a4509a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000974 1789674998.538662 write 0x02e063 thru 0x034e7d (0x6e1b bytes) 000975 1789674998.816543 trunc from 0x040000 to 0x032e5e (0xd1a1 bytes) 000976 1789674998.913795 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x32e5f): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e5f): Mirror 7: 0xc5287de4 (stale) Mirror 9: 0xc5287de4 (stale) Mirror 15: 0xc5287de4 (stale) Mirror 16: 0xc5287de4 (stale) Mirror 17: 0xc5287de4 (stale) Mirror 18: 0xc5287de4 (stale) Mirror 19: 0xc5287de4 (stale) Mirror 20: 0xc5287de4 (stale) Mirror 21: 0xc5287de4 (stale) Mirror 22: 0xc5287de4 (stale) Mirror 23: 0xc5287de4 (stale) Mirror 24: 0x65dc6f07 (stale) CRC-32 checksum value for chunk [0, 0x32e5f): Mirror 1: 0xe9a43b88 Mirror 25: 0xe9a43b88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e5f: skipped 000977 1789675000.648075 MIRROR_RESYNC 14 mirrors 000980 1789675003.561967 trunc from 0x032e5f to 0x01fe22 (0x1303c bytes) 000981 1789675003.806354 punch from 0x017ffa to 0x01d793 (0x579a bytes) 000982 1789675003.818127 write 0x0334d0 thru 0x03ffff (0xcb30 bytes) 000983 1789675003.839597 MIRROR_RESYNC 14 mirrors 000984 1789675007.486176 mapread 0x03538d thru 0x03e832 (0x94a6 bytes) 000986 1789675007.522795 read 0x016f1d thru 0x02028e (0x9372 bytes) 000987 1789675007.549831 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) 000988 1789675008.258519 trunc from 0x040000 to 0x02614a (0x19eb5 bytes) 000990 1789675008.387926 mapread 0x015a66 thru 0x0186cb (0x2c66 bytes) 000991 1789675008.422220 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2614b): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2614b): Mirror 7: 0x33581cf8 (stale) Mirror 9: 0x33581cf8 (stale) Mirror 15: 0x33581cf8 (stale) Mirror 17: 0x33581cf8 (stale) Mirror 18: 0x33581cf8 (stale) Mirror 19: 0x33581cf8 (stale) Mirror 20: 0x33581cf8 (stale) Mirror 21: 0x33581cf8 (stale) Mirror 22: 0x33581cf8 (stale) Mirror 23: 0x33581cf8 (stale) Mirror 24: 0x33581cf8 (stale) Mirror 25: 0x33581cf8 (stale) CRC-32 checksum value for chunk [0, 0x2614b): Mirror 1: 0x33581cf8 Mirror 26: 0x33581cf8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2614b: skipped 000992 1789675010.246188 write 0x00b03f thru 0x00d863 (0x2825 bytes) 000995 1789675010.382125 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2614b): Mirror 7: 0x33581cf8 (stale) Mirror 9: 0x33581cf8 (stale) Mirror 15: 0x33581cf8 (stale) Mirror 18: 0x33581cf8 (stale) Mirror 19: 0x33581cf8 (stale) Mirror 20: 0x33581cf8 (stale) Mirror 21: 0x33581cf8 (stale) Mirror 22: 0x33581cf8 (stale) Mirror 23: 0x33581cf8 (stale) Mirror 24: 0x33581cf8 (stale) Mirror 25: 0x33581cf8 (stale) Mirror 26: 0x33581cf8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2614b): Mirror 7: 0x33581cf8 (stale) Mirror 9: 0x33581cf8 (stale) Mirror 15: 0x33581cf8 (stale) Mirror 18: 0x33581cf8 (stale) Mirror 19: 0x33581cf8 (stale) Mirror 20: 0x33581cf8 (stale) Mirror 21: 0x33581cf8 (stale) Mirror 22: 0x33581cf8 (stale) Mirror 23: 0x33581cf8 (stale) Mirror 24: 0x33581cf8 (stale) Mirror 25: 0x33581cf8 (stale) Mirror 26: 0x33581cf8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2614b: skipped 000998 1789675011.297026 mapread 0x00941e thru 0x009ac4 (0x6a7 bytes) 000999 1789675011.311118 punch from 0x020be0 to 0x02614a (0x556b bytes) 001000 1789675011.319664 mapwrite 0x009a3f thru 0x00a6bf (0xc81 bytes) All operations completed A-OK! PASS 70a (181s) == sanity-flr test 71: check mirror extend parent fid ==== 15:56:58 (1789675018) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 15:57:03 (1789675023) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.351406 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267661 s, 3.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2be:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bb:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ba:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c0:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (15s) == sanity-flr test 200a: stress test ===================== 15:57:18 (1789675038) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8231968 .. Extending file size to 7070944 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8197344 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9095616 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8523232 .. Extending file size to 9245312 .. Extending file size to 9339648 .. Waiting 57481 57482 57485 57486 57488 done 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) 192.168.202.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:58:43 (1789675123) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4014144 .. Extending file size to 4076928 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8174368 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5241024 .. failed Extending file size to 7781376 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8340544 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9056544 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8266496 .. Waiting 58979 58981 58982 58983 58985 done 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) 192.168.202.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:00:09 (1789675209) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12423 s, 16.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.316863 s, 6.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xdb6d193e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:00:32 (1789675232) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173516 s, 12.1 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 16:00:42 (1789675242) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ed:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:00:51 (1789675251) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.283344 s, 7.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:01:01 (1789675261) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.280281 s, 7.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:01:10 (1789675270) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.346238 s, 6.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:01:47 (1789675307) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112281 s, 9.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:02:52 (1789675372) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.695521 s, 3.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (66s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:03:59 (1789675439) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. Creating new pool test_204f oleg250-server: Pool lustre.test_204f created Adding targets to pool oleg250-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg250-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.497818 s, 4.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2db:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg250-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg250-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg250-server: Pool lustre.test_204f destroyed PASS 204f (91s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:05:30 (1789675530) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:05:41 (1789675541) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:05:50 (1789675550) oleg250-server: Pool lustre.test_206 created Adding targets to pool oleg250-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg250-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg250-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg250-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg250-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:06:18 (1789675578) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113962 s, 9.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133976 s, 7.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x748ded83 Mirror 3: 0x748ded83 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:06:31 (1789675591) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:06:35 (1789675595) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:06:39 (1789675599) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.119221 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x7a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1a10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1a10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1a10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1450000 Read this (4 bytes): BBBB 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:07:20 (1789675640) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.018968 s, 432 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:07:27 (1789675647) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596017 s, 17.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x212 lmm_fid: [0x200000404:0x212:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x212 lmm_fid: [0x200000404:0x212:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834450 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-17 16:07:29.000000000 -0400 Modify: 2026-09-17 16:07:29.000000000 -0400 Change: 2026-09-17 16:07:31.000000000 -0400 Birth: 2026-09-17 16:07:29.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:07:37 (1789675657) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:07:41 (1789675661) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.82554 s, 49.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0680187 s, 60.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 17 16:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x214:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 17 16:07 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:07:51 (1789675671) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:08:00 (1789675680) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 2081 sec =========== 16:08:10 (1789675690) === sanity-flr: start cleanup 16:08:13 (1789675693) === === sanity-flr: finish cleanup 16:08:20 (1789675700) ===