-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 13:55:36 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg622-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:55:54 (1787853354) === oleg622-client.virtnet: executing check_config_client /mnt/lustre oleg622-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg622-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c2045485800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c2045485800.idle_timeout=debug disable quota as required oleg622-server: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:56:27 (1787853387) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7632] == sanity-flr test 0a: lfs mirror create with -N option == 13:56:29 (1787853389) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:56:46 (1787853406) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:56:48 (1787853408) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:56:51 (1787853411) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:57:17 (1787853437) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:57:20 (1787853440) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 13:57:23 (1787853443) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c2045485800.stats=clear osc.lustre-OST0001-osc-ffff9c2045485800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:57:30 (1787853450) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:57:40 (1787853460) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:57:47 (1787853467) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:57:50 (1787853470) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:57:57 (1787853477) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 13:58:07 (1787853487) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:58:15 (1787853495) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:58:18 (1787853498) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:58:25 (1787853505) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:58:35 (1787853515) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:58:42 (1787853522) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.544986 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13634 s, 7.7 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:58:52 (1787853532) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389764 s, 26.9 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:58:59 (1787853539) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:59:08 (1787853548) Stopping /mnt/lustre-ost1 (opts:) on oleg622-server oleg622-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2045485800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2045485800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg622-server: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg622-server: oleg622-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg622-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg622-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg622-server oleg622-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2045485800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2045485800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg622-server: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg622-server: oleg622-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg622-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (130s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:01:19 (1787853679) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.23627 s, 17.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg622-server oleg622-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2045485800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2045485800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg622-server: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg622-server: oleg622-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg622-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (38s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:01:57 (1787853717) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg622-server oleg622-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2045485800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2045485800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg622-server: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg622-server: oleg622-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg622-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg622-server oleg622-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2045485800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2045485800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg622-server: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg622-server: oleg622-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg622-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (65s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:03:02 (1787853782) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.94835 s, 13.3 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 14:03 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.832s user 0m0.000s sys 0m0.165s Stopping /mnt/lustre-ost1 (opts:) on oleg622-server oleg622-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c2045485800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c2045485800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.240s user 0m0.001s sys 0m0.191s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg622-server: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg622-server: oleg622-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg622-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg622-server oleg622-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c2045485800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2045485800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.326s user 0m0.000s sys 0m0.199s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg622-server: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg622-server: oleg622-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg622-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg622-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c2045485800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c2045485800.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:04:34 (1787853874) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:04:37 (1787853877) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:04:40 (1787853880) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 14:04:43 (1787853883) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 14:04:52 (1787853892) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.456877 s, 27.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:05:08 (1787853908) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357499 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416642 s, 25.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:05:15 (1787853915) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:05:23 (1787853923) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00147458 s, 694 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00112652 s, 909 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00198124 s, 517 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00226188 s, 453 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 14:05:35 (1787853935) create test files with size 15750925 .. ce99b75c72eb451be9a328e63a300831 - f18113cc27a1c414ada0d548e6a0b196 - 7f476bd7744df9ceae1495f09b855b23 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg622-client.virtnet /mnt/lustre (opts:) Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 14:05:58 (1787853958) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:06:15 (1787853975) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:06:22 (1787853982) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119178 s, 8.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333925 s, 31.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 14:06:32 (1787853992) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162663 s, 12.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.43872 s, 9.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.329984 s, 6.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 14:06:50 (1787854010) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 14:06:54 (1787854014) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:06:57 (1787854017) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448127 s, 23.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 14:07:05 (1787854025) llite.lustre-ffff9c204304d000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 14:07:13 (1787854033) llite.lustre-ffff9c204304d000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421217 s, 24.9 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334326 s, 31.4 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 14:07:22 (1787854042) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:07:26 (1787854046) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:07:37 (1787854057) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.217699 s, 48.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg622-client.virtnet /mnt/lustre (opts:) Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:07:51 (1787854071) llite.lustre-ffff9c204dcd1800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.249257 s, 42.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg622-client.virtnet /mnt/lustre (opts:) Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9c205471f000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:08:06 (1787854086) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (48s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:08:54 (1787854134) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:09:35 (1787854175) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 14:09:39 (1787854179) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.306306 s, 10.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.150809 s, 20.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xba40b89d (nosync,stale) Mirror 2: 0xba40b89d (nosync,stale) ** verify mirror 2 doesn't change original checksum: dca45949577f07b56fd3de5aeb558a24 - mirror 1 checksum: f547d57c838cdb0fec242b706a8430ac - mirror 2 checksum: dca45949577f07b56fd3de5aeb558a24 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:09:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:09:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: dca45949577f07b56fd3de5aeb558a24 - mirror 1 checksum: f547d57c838cdb0fec242b706a8430ac - mirror 2 checksum: f547d57c838cdb0fec242b706a8430ac - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:09:41-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:09:41-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf0:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:09:53 (1787854193) Failing mds1 on oleg622-server Stopping /mnt/lustre-mds1 (opts:) on oleg622-server 14:10:01 (1787854201) shut down facet: mds1 facet_host: oleg622-server facet_failover_host: oleg622-server Failover mds1 to oleg622-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg622-server: oleg622-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg622-client: oleg622-server: ssh exited with exit code 1 Started lustre-MDT0000 14:10:25 (1787854225) targets are mounted 14:10:25 (1787854225) facet_failover done oleg622-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535051 s, 19.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf1:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:10:42 (1787854242) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:10:46 (1787854246) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:10:49 (1787854249) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00622623 s, 2.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:10:57 (1787854257) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:11:00 (1787854260) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00397247 s, 1.0 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.128625 s, 31.8 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:11:08 (1787854268) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:11:32 (1787854292) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:12:00 (1787854320) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:12:26 (1787854346) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:12:51 (1787854371) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 14:12:58 (1787854378) mirror create pid 55596 mirror split pid 55597 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55596 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55597 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 14:14:06 (1787854446) Seed set to 4448 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1787854448.625147 write 0x007a61 thru 0x009322 (0x18c2 bytes) 000003 1787854448.659315 read 0x003568 thru 0x004716 (0x11af bytes) 000004 1787854448.674876 mapread 0x007d81 thru 0x009322 (0x15a2 bytes) 000005 1787854448.677147 read 0x006f18 thru 0x009322 (0x240b bytes) 000006 1787854448.702087 mapwrite 0x011686 thru 0x020dc1 (0xf73c bytes) 000007 1787854448.894227 mapwrite 0x0144ca thru 0x01f7f6 (0xb32d bytes) 000009 1787854448.944500 write 0x0398cd thru 0x03ffff (0x6733 bytes) 000011 1787854448.947352 mapwrite 0x00d91f thru 0x00ea8e (0x1170 bytes) 000014 1787854449.078606 mapwrite 0x024b4b thru 0x02b8bc (0x6d72 bytes) 000017 1787854449.222940 mapwrite 0x025ce6 thru 0x02b8e3 (0x5bfe bytes) 000019 1787854449.313967 write 0x011cbd thru 0x012667 (0x9ab bytes) truncating to largest ever: 0xf38e 000021 1787854449.315419 trunc from 0x040000 to 0x00f38d (0x30c72 bytes) 000022 1787854449.350935 write 0x021dff thru 0x02afe9 (0x91eb bytes) 000024 1787854449.356543 trunc from 0x02afea to 0x0061f5 (0x24df4 bytes) 000028 1787854449.400963 mapwrite 0x028c42 thru 0x035a0c (0xcdcb bytes) 000030 1787854449.607913 trunc from 0x035a0d to 0x00afe6 (0x2aa26 bytes) 000032 1787854449.657950 trunc from 0x00afe7 to 0x009d9b (0x124b bytes) truncating to largest ever: 0x17b21 000034 1787854449.678156 trunc from 0x009d9c to 0x017b20 (0xdd85 bytes) 000036 1787854449.710169 write 0x03c90c thru 0x03ffff (0x36f4 bytes) 000039 1787854449.713235 mapread 0x037b8a thru 0x03e88f (0x6d06 bytes) 000040 1787854449.737814 read 0x01d99a thru 0x0292ee (0xb955 bytes) 000045 1787854449.811473 MIRROR_EXTEND to 2 mirrors 000046 1787854451.097346 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000047 1787854451.336884 write 0x017971 thru 0x01b0ca (0x375a bytes) 000049 1787854451.366524 mapread 0x034735 thru 0x03ffff (0xb8cb bytes) 000051 1787854451.407292 mapread 0x01f5ef thru 0x022f9b (0x39ad bytes) 000053 1787854451.431680 read 0x00efd1 thru 0x01dbf0 (0xec20 bytes) 000056 1787854451.449161 MIRROR_EXTEND to 2 mirrors 000058 1787854452.687977 mapwrite 0x0270e9 thru 0x030d76 (0x9c8e bytes) 000061 1787854452.903640 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9f3fd760 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb5aeb1e6 Mirror 3: 0xb5aeb1e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000062 1787854454.142075 mapread 0x0059c5 thru 0x015796 (0xfdd2 bytes) 000067 1787854454.184856 read 0x01d456 thru 0x02381b (0x63c6 bytes) 000068 1787854454.198520 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9f3fd760 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9f3fd760 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000069 1787854454.573419 mapwrite 0x01cc8a thru 0x022f1e (0x6295 bytes) truncating to largest ever: 0x3be24 000072 1787854454.661337 trunc from 0x040000 to 0x03be23 (0x41dc bytes) 000073 1787854454.700973 write 0x03dbda thru 0x03ffff (0x2426 bytes) 000076 1787854454.717754 MIRROR_RESYNC 2 mirrors 000078 1787854455.633963 read 0x01276e thru 0x01cd18 (0xa5ab bytes) 000079 1787854455.659963 trunc from 0x040000 to 0x0333f6 (0xcc09 bytes) 000080 1787854455.750994 mapread 0x001e8e thru 0x00ae36 (0x8fa9 bytes) 000081 1787854455.772874 mapwrite 0x0340e4 thru 0x03d9fd (0x991a bytes) 000083 1787854455.865138 mapread 0x023ced thru 0x032e28 (0xf13c bytes) 000084 1787854455.878848 mapread 0x028441 thru 0x0350ee (0xccae bytes) 000085 1787854455.884178 MIRROR_RESYNC 2 mirrors 000086 1787854456.479418 trunc from 0x03d9fe to 0x007d48 (0x35cb5 bytes) 000087 1787854456.614245 mapread 0x001c86 thru 0x007d48 (0x60c3 bytes) 000088 1787854456.630906 write 0x02e780 thru 0x0338ba (0x513b bytes) 000091 1787854456.633913 mapwrite 0x005059 thru 0x00ea47 (0x99ef bytes) 000093 1787854456.692407 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000096 1787854456.934670 trunc from 0x0338bb to 0x006495 (0x2d425 bytes) 000098 1787854456.990590 MIRROR_EXTEND to 2 mirrors 000100 1787854457.829054 MIRROR_EXTEND to 3 mirrors truncating to largest ever: 0x3e9a0 000103 1787854459.004469 trunc from 0x006496 to 0x03e99f (0x3850a bytes) 000104 1787854459.121265 mapread 0x0321ad thru 0x03e99f (0xc7f3 bytes) 000105 1787854459.164409 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9a0): Mirror 3: 0xa9c98f1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9a0): Mirror 3: 0xa9c98f1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9a0: skipped 000106 1787854459.626955 read 0x0033a9 thru 0x0098b5 (0x650d bytes) 000109 1787854459.642425 write 0x023056 thru 0x023e92 (0xe3d bytes) 000110 1787854459.664874 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000111 1787854459.852639 mapread 0x024b8d thru 0x02fed2 (0xb346 bytes) 000114 1787854459.884582 mapwrite 0x00d27d thru 0x00e7e9 (0x156d bytes) 000115 1787854459.948557 read 0x02691d thru 0x02e021 (0x7705 bytes) 000116 1787854459.949167 write 0x0178f6 thru 0x01c32d (0x4a38 bytes) 000117 1787854459.960081 write 0x016767 thru 0x01a1d4 (0x3a6e bytes) 000118 1787854459.966172 read 0x01b019 thru 0x0222f5 (0x72dd bytes) 000119 1787854459.979224 trunc from 0x03e9a0 to 0x023221 (0x1b77e bytes) 000120 1787854460.009286 mapwrite 0x0184d5 thru 0x02493a (0xc466 bytes) 000121 1787854460.116840 mapread 0x01b67e thru 0x02469c (0x901f bytes) 000124 1787854460.122245 write 0x03889c thru 0x03ac89 (0x23ee bytes) 000126 1787854460.127227 write 0x01b8f8 thru 0x01d17a (0x1883 bytes) 000127 1787854460.131008 mapread 0x030c34 thru 0x03ac89 (0xa056 bytes) 000129 1787854460.174107 mapread 0x026198 thru 0x02f7be (0x9627 bytes) 000130 1787854460.239474 read 0x03913c thru 0x03ac89 (0x1b4e bytes) 000132 1787854460.242949 mapread 0x02a357 thru 0x033fe8 (0x9c92 bytes) 000135 1787854460.247904 write 0x02e98a thru 0x038837 (0x9eae bytes) 000137 1787854460.251063 mapread 0x004918 thru 0x0096f5 (0x4dde bytes) 000140 1787854460.255197 trunc from 0x03ac8a to 0x020460 (0x1a829 bytes) 000141 1787854460.275747 read 0x00489f thru 0x0111a4 (0xc906 bytes) 000142 1787854460.276740 mapwrite 0x014b9a thru 0x016c53 (0x20ba bytes) 000143 1787854460.392918 read 0x00482f thru 0x00cf21 (0x86f3 bytes) 000145 1787854460.396925 trunc from 0x020461 to 0x018bc4 (0x789c bytes) 000147 1787854460.422128 write 0x031491 thru 0x03ffff (0xeb6f bytes) 000148 1787854460.425653 trunc from 0x040000 to 0x004c3b (0x3b3c4 bytes) 000153 1787854460.455042 write 0x027e30 thru 0x0286d7 (0x8a8 bytes) 000156 1787854460.456914 write 0x00dc2a thru 0x010caf (0x3086 bytes) 000158 1787854460.476092 read 0x009300 thru 0x00d414 (0x4115 bytes) 000160 1787854460.477290 mapwrite 0x02eed7 thru 0x03d853 (0xe97d bytes) truncating to largest ever: 0x3ef16 000163 1787854460.650081 trunc from 0x03d854 to 0x03ef15 (0x16c2 bytes) 000174 1787854460.688569 write 0x027a49 thru 0x02c396 (0x494e bytes) 000175 1787854460.689683 mapread 0x0292b7 thru 0x03727b (0xdfc5 bytes) 000178 1787854460.695356 trunc from 0x03ef16 to 0x01822a (0x26ceb bytes) 000179 1787854460.716749 mapread 0x00d5a7 thru 0x01822a (0xac84 bytes) 000180 1787854460.717776 write 0x01260c thru 0x017375 (0x4d6a bytes) 000181 1787854460.718769 mapread 0x006c84 thru 0x008990 (0x1d0d bytes) 000182 1787854460.719667 mapread 0x015ed6 thru 0x017951 (0x1a7c bytes) 000184 1787854460.720941 read 0x001db5 thru 0x00a5f2 (0x883e bytes) 000185 1787854460.721492 read 0x00af46 thru 0x015aaa (0xab65 bytes) 000186 1787854460.723138 read 0x007908 thru 0x00f2a2 (0x799b bytes) 000187 1787854460.723825 write 0x017a3f thru 0x024b52 (0xd114 bytes) 000188 1787854460.725557 mapwrite 0x0280d6 thru 0x02a455 (0x2380 bytes) 000190 1787854460.885868 trunc from 0x02a456 to 0x03e514 (0x140bf bytes) 000192 1787854460.915053 mapwrite 0x00f7c8 thru 0x0148c8 (0x5101 bytes) 000194 1787854461.027111 write 0x0318c8 thru 0x03960f (0x7d48 bytes) 000198 1787854461.053993 mapread 0x000b8c thru 0x007823 (0x6c98 bytes) 000200 1787854461.055818 trunc from 0x03e515 to 0x012c05 (0x2b90f bytes) 000204 1787854461.130783 mapread 0x011d90 thru 0x012c05 (0xe76 bytes) 000205 1787854461.134007 mapwrite 0x000388 thru 0x00dfd9 (0xdc52 bytes) 000207 1787854461.241208 read 0x005533 thru 0x012c05 (0xd6d3 bytes) 000209 1787854461.243339 mapread 0x0035fa thru 0x00ac52 (0x7659 bytes) 000212 1787854461.254077 MIRROR_EXTEND to 2 mirrors 000213 1787854461.988200 MIRROR_RESYNC 2 mirrors 000214 1787854462.659109 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000215 1787854462.882286 mapread 0x00d2e4 thru 0x012c05 (0x5922 bytes) 000217 1787854462.924899 read 0x011e4e thru 0x012c05 (0xdb8 bytes) 000218 1787854462.926256 read 0x000d47 thru 0x00c9aa (0xbc64 bytes) 000219 1787854462.942487 mapwrite 0x02818d thru 0x0305fc (0x8470 bytes) 000220 1787854463.201494 mapread 0x0228f1 thru 0x02fdc4 (0xd4d4 bytes) 000221 1787854463.203844 mapread 0x0034c7 thru 0x00cfed (0x9b27 bytes) 000223 1787854463.300463 trunc from 0x0305fd to 0x018e8d (0x1776f bytes) 000226 1787854463.333502 mapwrite 0x03e0fe thru 0x03ffff (0x1f02 bytes) 000231 1787854463.477980 mapread 0x031293 thru 0x034e55 (0x3bc3 bytes) 000234 1787854463.481779 mapwrite 0x0357ea thru 0x039374 (0x3b8b bytes) 000240 1787854463.562575 mapwrite 0x026e27 thru 0x029bad (0x2d87 bytes) 000241 1787854463.774996 MIRROR_EXTEND to 2 mirrors 000242 1787854464.570002 mapwrite 0x035652 thru 0x03c135 (0x6ae4 bytes) 000244 1787854464.853807 mapwrite 0x009551 thru 0x00bf41 (0x29f1 bytes) 000246 1787854464.950586 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000247 1787854465.264907 trunc from 0x040000 to 0x003245 (0x3cdba bytes) 000248 1787854465.292741 write 0x03d656 thru 0x03ffff (0x29aa bytes) 000249 1787854465.295344 write 0x024d0d thru 0x02f6f5 (0xa9e9 bytes) 000250 1787854465.315894 write 0x00b3cf thru 0x00b8bb (0x4ed bytes) 000252 1787854465.339177 mapread 0x016d10 thru 0x01abc2 (0x3eb3 bytes) 000253 1787854465.397542 read 0x03d4a0 thru 0x03e5fe (0x115f bytes) 000255 1787854465.398862 trunc from 0x040000 to 0x0077a7 (0x38858 bytes) 000256 1787854465.426127 write 0x005db9 thru 0x011374 (0xb5bc bytes) 000257 1787854465.435215 read 0x010d2f thru 0x011374 (0x646 bytes) 000258 1787854465.436134 mapwrite 0x03cb80 thru 0x03ffff (0x3480 bytes) 000259 1787854465.603004 read 0x00df3e thru 0x01064f (0x2712 bytes) 000261 1787854465.606953 mapread 0x000a97 thru 0x002def (0x2359 bytes) 000263 1787854465.611442 trunc from 0x040000 to 0x01616c (0x29e93 bytes) 000264 1787854465.638257 mapread 0x00fa63 thru 0x011840 (0x1dde bytes) 000267 1787854465.640079 trunc from 0x01616d to 0x035ac1 (0x1f955 bytes) 000269 1787854465.678004 mapread 0x02885a thru 0x02f69b (0x6e42 bytes) 000272 1787854465.695002 mapread 0x01f6ca thru 0x02d0af (0xd9e6 bytes) 000274 1787854465.698581 mapread 0x02d39d thru 0x030b6d (0x37d1 bytes) 000276 1787854465.701115 mapread 0x027c9b thru 0x035ac1 (0xde27 bytes) 000277 1787854465.703399 write 0x00f23a thru 0x01c1ab (0xcf72 bytes) 000278 1787854465.706017 trunc from 0x035ac2 to 0x0099b7 (0x2c10a bytes) 000279 1787854465.729598 write 0x0186d8 thru 0x01c086 (0x39af bytes) 000287 1787854465.737847 mapwrite 0x00054d thru 0x005ede (0x5992 bytes) 000290 1787854465.839063 mapwrite 0x003f8c thru 0x010493 (0xc508 bytes) 000292 1787854466.060813 read 0x00ded3 thru 0x015b58 (0x7c86 bytes) 000294 1787854466.075828 mapread 0x013528 thru 0x01c086 (0x8b5f bytes) 000297 1787854466.081060 write 0x03d7df thru 0x03ffff (0x2821 bytes) 000298 1787854466.082252 write 0x033142 thru 0x0350e4 (0x1fa3 bytes) 000299 1787854466.091475 read 0x01b0a3 thru 0x01e975 (0x38d3 bytes) 000301 1787854466.093054 mapwrite 0x027af9 thru 0x02dc65 (0x616d bytes) 000303 1787854466.150957 mapread 0x017206 thru 0x01c727 (0x5522 bytes) 000305 1787854466.153999 read 0x02aca5 thru 0x0314b5 (0x6811 bytes) 000307 1787854466.155997 mapwrite 0x01f8f8 thru 0x021eff (0x2608 bytes) 000308 1787854466.233850 trunc from 0x040000 to 0x020629 (0x1f9d6 bytes) 000315 1787854466.259039 trunc from 0x02062a to 0x01e876 (0x1db3 bytes) 000316 1787854466.294173 read 0x001829 thru 0x00bf38 (0xa710 bytes) 000317 1787854466.294974 write 0x01f0dc thru 0x02b4b9 (0xc3de bytes) 000318 1787854466.297115 mapwrite 0x02aac2 thru 0x037854 (0xcd93 bytes) 000319 1787854466.469859 read 0x013bc4 thru 0x01c5ac (0x89e9 bytes) 000320 1787854466.472085 mapread 0x01e8c5 thru 0x022809 (0x3f45 bytes) 000321 1787854466.473063 trunc from 0x037855 to 0x0336e1 (0x4173 bytes) 000324 1787854466.519833 trunc from 0x0336e2 to 0x03580f (0x212e bytes) 000331 1787854466.566539 mapwrite 0x02e2e9 thru 0x0343cb (0x60e3 bytes) 000332 1787854466.657261 write 0x005f51 thru 0x006b11 (0xbc1 bytes) 000333 1787854466.658148 trunc from 0x035810 to 0x03c2ec (0x6add bytes) 000336 1787854466.677652 read 0x02f19e thru 0x037357 (0x81ba bytes) 000339 1787854466.694437 read 0x023b75 thru 0x028d76 (0x5202 bytes) 000340 1787854466.695583 mapwrite 0x03e59e thru 0x03ffff (0x1a62 bytes) 000343 1787854466.771787 mapwrite 0x010529 thru 0x01fccd (0xf7a5 bytes) 000345 1787854466.870945 read 0x036ceb thru 0x03ffff (0x9315 bytes) 000346 1787854466.871520 mapwrite 0x007a9e thru 0x015934 (0xde97 bytes) 000349 1787854466.945993 write 0x0046d7 thru 0x00f05d (0xa987 bytes) 000350 1787854466.947938 write 0x02f893 thru 0x036e06 (0x7574 bytes) 000353 1787854466.950987 mapwrite 0x01604e thru 0x02130a (0xb2bd bytes) 000355 1787854467.020499 trunc from 0x040000 to 0x00c029 (0x33fd6 bytes) 000357 1787854467.059596 write 0x02b73a thru 0x02bc79 (0x540 bytes) 000358 1787854467.060524 write 0x033f71 thru 0x034942 (0x9d2 bytes) 000360 1787854467.061875 trunc from 0x034943 to 0x024ae6 (0xfe5c bytes) 000361 1787854467.086335 mapread 0x022dc6 thru 0x024ae6 (0x1d21 bytes) 000362 1787854467.104283 mapwrite 0x021297 thru 0x022fd4 (0x1d3e bytes) 000363 1787854467.184035 MIRROR_EXTEND to 2 mirrors 000364 1787854468.257761 write 0x01024b thru 0x010c18 (0x9ce bytes) 000366 1787854468.360915 read 0x015e12 thru 0x024ae6 (0xecd5 bytes) 000367 1787854468.378221 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24ae7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ae7): Mirror 2: 0xe7db5567 (stale) CRC-32 checksum value for chunk [0, 0x24ae7): Mirror 1: 0xae14718f Mirror 3: 0xae14718f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ae7: skipped 000368 1787854469.394051 write 0x019f15 thru 0x021385 (0x7471 bytes) 000369 1787854469.493597 trunc from 0x024ae7 to 0x02b3ee (0x6908 bytes) 000370 1787854469.561050 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b3ef): Mirror 3: 0xae2ac0e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b3ef): Mirror 3: 0xae2ac0e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b3ef: skipped 000375 1787854469.944830 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2b3ef): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b3ef): Mirror 3: 0xae2ac0e4 (stale) CRC-32 checksum value for chunk [0, 0x2b3ef): Mirror 1: 0x6b76f2a4 Mirror 4: 0x6b76f2a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b3ef: skipped 000376 1787854471.270245 trunc from 0x02b3ef to 0x005d3a (0x256b4 bytes) 000377 1787854471.393899 mapread 0x000553 thru 0x005d3a (0x57e8 bytes) 000378 1787854471.423660 mapwrite 0x02c5eb thru 0x030dd7 (0x47ed bytes) 000380 1787854471.566101 mapwrite 0x00d078 thru 0x017d73 (0xacfc bytes) 000381 1787854471.679501 write 0x01a4a0 thru 0x023490 (0x8ff1 bytes) 000382 1787854471.696109 read 0x011fc8 thru 0x012d51 (0xd8a bytes) 000383 1787854471.710201 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30dd8): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30dd8): Mirror 3: 0xe55de64a (stale) Mirror 4: 0x437048db (stale) CRC-32 checksum value for chunk [0, 0x30dd8): Mirror 1: 0xe6e1d711 Mirror 5: 0xe6e1d711 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30dd8: skipped 000384 1787854473.472164 mapread 0x002567 thru 0x0057bf (0x3259 bytes) 000385 1787854473.548901 MIRROR_RESYNC 4 mirrors 000388 1787854474.849568 read 0x02641a thru 0x027c45 (0x182c bytes) 000390 1787854474.855862 mapread 0x00615a thru 0x01598c (0xf833 bytes) 000392 1787854474.887111 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000393 1787854475.273815 write 0x039ffb thru 0x03ffff (0x6005 bytes) 000394 1787854475.360479 mapwrite 0x002e58 thru 0x00304a (0x1f3 bytes) 000395 1787854475.436855 mapread 0x016bd6 thru 0x018c7a (0x20a5 bytes) 000397 1787854475.445084 read 0x036096 thru 0x03ffff (0x9f6a bytes) 000398 1787854475.455210 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x54be7e1a (stale) Mirror 5: 0x54be7e1a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfe36bbc6 Mirror 6: 0xfe36bbc6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000400 1787854476.164441 MIRROR_RESYNC 4 mirrors 000401 1787854477.593400 trunc from 0x040000 to 0x037ec5 (0x813a bytes) 000403 1787854477.709854 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ec6): Mirror 4: 0x22b3fed8 (stale) Mirror 6: 0x22b3fed8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ec6): Mirror 4: 0x22b3fed8 (stale) Mirror 6: 0x22b3fed8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ec6: skipped 000407 1787854478.055277 write 0x0217e1 thru 0x0260e0 (0x4900 bytes) 000409 1787854478.070598 write 0x037355 thru 0x037629 (0x2d5 bytes) 000410 1787854478.080917 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37ec6): Mirror 4: 0x22b3fed8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37ec6): Mirror 4: 0x22b3fed8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37ec6: skipped 000411 1787854478.542080 read 0x01c130 thru 0x022dd1 (0x6ca2 bytes) 000412 1787854478.552879 trunc from 0x037ec6 to 0x01592b (0x2259a bytes) 000413 1787854478.575367 mapwrite 0x006003 thru 0x00e037 (0x8035 bytes) 000417 1787854478.685970 trunc from 0x01592c to 0x00cf43 (0x89e8 bytes) 000422 1787854478.712905 read 0x003796 thru 0x0064a4 (0x2d0f bytes) 000423 1787854478.713866 trunc from 0x00cf44 to 0x015557 (0x8614 bytes) 000424 1787854478.738735 trunc from 0x015558 to 0x00f39f (0x61b8 bytes) 000425 1787854478.762952 mapread 0x00773e thru 0x00f39f (0x7c62 bytes) 000426 1787854478.776876 mapread 0x003dc3 thru 0x006a1e (0x2c5c bytes) 000428 1787854478.778949 write 0x030d78 thru 0x03458e (0x3817 bytes) 000429 1787854478.780632 write 0x020091 thru 0x02d74a (0xd6ba bytes) 000431 1787854478.798790 read 0x02b315 thru 0x02e0bd (0x2da9 bytes) 000432 1787854478.800159 mapwrite 0x01618f thru 0x01e6d8 (0x854a bytes) 000433 1787854478.871108 write 0x01919c thru 0x01d062 (0x3ec7 bytes) 000434 1787854478.876763 trunc from 0x03458f to 0x023786 (0x10e08 bytes) 000437 1787854478.900867 write 0x03b716 thru 0x03cb9b (0x1486 bytes) 000438 1787854478.902193 mapread 0x021d15 thru 0x031c06 (0xfef2 bytes) 000441 1787854478.916851 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000442 1787854479.176608 trunc from 0x03cb9c to 0x03c616 (0x585 bytes) 000444 1787854479.195862 write 0x03f949 thru 0x03ffff (0x6b7 bytes) 000445 1787854479.198206 write 0x025765 thru 0x030220 (0xaabc bytes) 000446 1787854479.209841 mapwrite 0x00a2ce thru 0x0135ae (0x92e1 bytes) 000448 1787854479.280773 read 0x018983 thru 0x01af63 (0x25e1 bytes) 000449 1787854479.286473 read 0x03b532 thru 0x03ffff (0x4ace bytes) 000450 1787854479.296388 trunc from 0x040000 to 0x028ba7 (0x17458 bytes) 000451 1787854479.318373 write 0x0023f2 thru 0x00d059 (0xac68 bytes) 000456 1787854479.322192 write 0x00802e thru 0x016b3a (0xeb0d bytes) 000458 1787854479.325938 mapwrite 0x015c53 thru 0x017efd (0x22ab bytes) 000460 1787854479.384440 mapwrite 0x025f84 thru 0x028703 (0x2780 bytes) 000461 1787854479.430302 mapread 0x017905 thru 0x01d32f (0x5a2b bytes) 000462 1787854479.431544 mapread 0x024e42 thru 0x028ba7 (0x3d66 bytes) 000469 1787854479.437009 write 0x031e2b thru 0x035f18 (0x40ee bytes) 000470 1787854479.438312 read 0x012dfb thru 0x01ec33 (0xbe39 bytes) 000472 1787854479.439986 read 0x01ef0e thru 0x029370 (0xa463 bytes) 000474 1787854479.446790 mapread 0x010587 thru 0x012823 (0x229d bytes) 000475 1787854479.447576 trunc from 0x035f19 to 0x001cb8 (0x34260 bytes) 000476 1787854479.467194 write 0x030d8c thru 0x031ea2 (0x1117 bytes) 000479 1787854479.472388 read 0x00dcf4 thru 0x01943a (0xb747 bytes) 000481 1787854479.485316 trunc from 0x031ea3 to 0x009410 (0x28a92 bytes) 000483 1787854479.511332 mapread 0x000f95 thru 0x009410 (0x847c bytes) 000488 1787854479.516173 mapwrite 0x020a85 thru 0x02feff (0xf47b bytes) 000492 1787854479.703396 write 0x0028e8 thru 0x00ef5a (0xc673 bytes) 000493 1787854479.710992 read 0x0109a5 thru 0x01cc2b (0xc287 bytes) 000494 1787854479.724147 write 0x01add1 thru 0x01d1c8 (0x23f8 bytes) 000496 1787854479.725960 write 0x035e95 thru 0x036323 (0x48f bytes) 000497 1787854479.729764 write 0x01c64c thru 0x029cef (0xd6a4 bytes) 000498 1787854479.732043 write 0x03cb7e thru 0x03ffff (0x3482 bytes) 000500 1787854479.743817 MIRROR_EXTEND to 2 mirrors 000503 1787854480.665405 MIRROR_EXTEND to 3 mirrors 000504 1787854481.543326 write 0x00ba08 thru 0x01389e (0x7e97 bytes) 000505 1787854481.632433 mapwrite 0x02a05a thru 0x035681 (0xb628 bytes) 000507 1787854481.745178 write 0x02111c thru 0x023cf1 (0x2bd6 bytes) 000508 1787854481.764886 mapread 0x02b675 thru 0x03045e (0x4dea bytes) 000509 1787854481.766882 trunc from 0x040000 to 0x0195fc (0x26a03 bytes) 000510 1787854481.823033 trunc from 0x0195fd to 0x03db09 (0x2450d bytes) 000512 1787854481.858053 mapwrite 0x03ab12 thru 0x03ffff (0x54ee bytes) 000514 1787854482.023607 write 0x004c4f thru 0x00884f (0x3c01 bytes) 000515 1787854482.056465 trunc from 0x040000 to 0x01ae90 (0x2516f bytes) 000516 1787854482.095057 read 0x01a98b thru 0x01ae90 (0x506 bytes) 000517 1787854482.108265 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ae91): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ae91): Mirror 2: 0x89fd5cf1 (stale) Mirror 3: 0x89fd5cf1 (stale) CRC-32 checksum value for chunk [0, 0x1ae91): Mirror 1: 0xabca9620 Mirror 4: 0xabca9620 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ae91: skipped 000518 1787854483.427277 read 0x019c52 thru 0x01ae90 (0x123f bytes) 000519 1787854483.444322 write 0x03b874 thru 0x03ffff (0x478c bytes) 000520 1787854483.617327 mapread 0x034cae thru 0x03ffff (0xb352 bytes) 000522 1787854483.679085 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd36cc3cb (stale) Mirror 4: 0x86d4d20f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xd36cc3cb (stale) Mirror 4: 0x86d4d20f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000523 1787854484.141879 write 0x0140b1 thru 0x0232cc (0xf21c bytes) 000524 1787854484.163937 mapread 0x0391d4 thru 0x039fc7 (0xdf4 bytes) 000525 1787854484.174328 write 0x009456 thru 0x01530b (0xbeb6 bytes) 000528 1787854484.189098 read 0x03f4f0 thru 0x03ffff (0xb10 bytes) 000530 1787854484.203986 mapread 0x01dae7 thru 0x02aa83 (0xcf9d bytes) 000531 1787854484.221826 trunc from 0x040000 to 0x02fa86 (0x10579 bytes) 000534 1787854484.275078 mapread 0x00f57d thru 0x019fbf (0xaa43 bytes) 000535 1787854484.276617 mapread 0x0108d6 thru 0x01af78 (0xa6a3 bytes) 000536 1787854484.277982 mapread 0x025922 thru 0x028e0b (0x34ea bytes) 000537 1787854484.279076 write 0x02173d thru 0x027974 (0x6238 bytes) 000538 1787854484.280870 write 0x0279a6 thru 0x031a17 (0xa072 bytes) 000539 1787854484.282216 trunc from 0x031a18 to 0x031502 (0x515 bytes) 000541 1787854484.314880 mapread 0x018f60 thru 0x02061f (0x76c0 bytes) 000542 1787854484.315785 trunc from 0x031503 to 0x03459e (0x309c bytes) 000543 1787854484.344116 trunc from 0x03459f to 0x021ab2 (0x12aec bytes) 000544 1787854484.368000 mapwrite 0x023550 thru 0x031cd7 (0xe788 bytes) 000545 1787854484.466724 mapwrite 0x02ab14 thru 0x0307f6 (0x5ce3 bytes) 000546 1787854484.527882 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31cd8): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31cd8): Mirror 2: 0x53f8616a (stale) Mirror 4: 0x2af99138 (stale) CRC-32 checksum value for chunk [0, 0x31cd8): Mirror 1: 0xc6dd3a8c Mirror 5: 0xc6dd3a8c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31cd8: skipped 000547 1787854485.602307 mapread 0x00d6c2 thru 0x012040 (0x497f bytes) 000549 1787854485.624932 mapwrite 0x039626 thru 0x03eeb1 (0x588c bytes) 000551 1787854485.874669 mapread 0x012cf1 thru 0x014e1d (0x212d bytes) 000552 1787854485.884281 write 0x0317b1 thru 0x03b076 (0x98c6 bytes) 000553 1787854485.890129 trunc from 0x03eeb2 to 0x0379ac (0x7505 bytes) 000554 1787854485.917113 mapwrite 0x01c272 thru 0x026a01 (0xa790 bytes) 000557 1787854486.040026 read 0x001680 thru 0x00bac2 (0xa443 bytes) 000559 1787854486.072150 mapwrite 0x00ff3a thru 0x0130b1 (0x3178 bytes) 000560 1787854486.186339 mapwrite 0x03d18a thru 0x03ffff (0x2e76 bytes) 000562 1787854486.295453 mapwrite 0x013a2c thru 0x016fab (0x3580 bytes) 000563 1787854486.364981 read 0x01590e thru 0x01c576 (0x6c69 bytes) 000564 1787854486.365762 read 0x024982 thru 0x0252c3 (0x942 bytes) 000565 1787854486.367404 mapwrite 0x018bae thru 0x028a3c (0xfe8f bytes) 000566 1787854486.466482 mapwrite 0x03e0d5 thru 0x03ffff (0x1f2b bytes) 000567 1787854486.518436 read 0x01bb10 thru 0x027aae (0xbf9f bytes) 000568 1787854486.519718 mapread 0x0139ac thru 0x020259 (0xc8ae bytes) 000569 1787854486.521545 write 0x0361c5 thru 0x03ffff (0x9e3b bytes) 000570 1787854486.523951 trunc from 0x040000 to 0x0010ae (0x3ef51 bytes) 000573 1787854486.559831 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x10af): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10af): Mirror 2: 0x41ca0ef9 (stale) Mirror 4: 0x41ca0ef9 (stale) Mirror 5: 0x41ca0ef9 (stale) CRC-32 checksum value for chunk [0, 0x10af): Mirror 1: 0x41ca0ef9 Mirror 6: 0x41ca0ef9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10af: skipped 000576 1787854487.379137 read 0x000064 thru 0x0010ae (0x104b bytes) 000577 1787854487.385087 write 0x008783 thru 0x00ebb3 (0x6431 bytes) 000579 1787854487.526860 mapread 0x00e67d thru 0x00ebb3 (0x537 bytes) 000581 1787854487.539972 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xebb4): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xebb4): Mirror 2: 0x1299eee8 (stale) Mirror 4: 0xf93be56b (stale) Mirror 5: 0x682c6380 (stale) Mirror 6: 0x3f558c42 (stale) CRC-32 checksum value for chunk [0, 0xebb4): Mirror 1: 0xc3cd20ed Mirror 7: 0xc3cd20ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xebb4: skipped 000582 1787854488.918908 mapwrite 0x000fa4 thru 0x004913 (0x3970 bytes) 000584 1787854489.218722 write 0x039d57 thru 0x03ffff (0x62a9 bytes) 000585 1787854489.220761 trunc from 0x040000 to 0x0191c9 (0x26e36 bytes) 000588 1787854489.247658 write 0x009b45 thru 0x009f54 (0x410 bytes) 000589 1787854489.249109 mapwrite 0x00b63f thru 0x019b7a (0xe53c bytes) 000590 1787854489.346646 write 0x00fde6 thru 0x01b2d2 (0xb4ed bytes) 000591 1787854489.350565 read 0x00e68c thru 0x012ac6 (0x443b bytes) 000592 1787854489.354934 mapread 0x009a3c thru 0x01217e (0x8743 bytes) 000593 1787854489.360323 write 0x005e6b thru 0x0070be (0x1254 bytes) 000594 1787854489.379969 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b2d3): Mirror 4: 0xb8a44f8 (stale) Mirror 5: 0xc8a3e825 (stale) Mirror 6: 0x98ecb1ab (stale) Mirror 7: 0x77beb47b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b2d3): Mirror 4: 0xb8a44f8 (stale) Mirror 5: 0xc8a3e825 (stale) Mirror 6: 0x98ecb1ab (stale) Mirror 7: 0x77beb47b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b2d3: skipped 000595 1787854489.824617 mapread 0x00343c thru 0x009f0b (0x6ad0 bytes) 000597 1787854489.847889 write 0x027119 thru 0x030bdf (0x9ac7 bytes) 000598 1787854489.853562 write 0x035300 thru 0x03ffff (0xad00 bytes) 000599 1787854489.864985 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xfe155ae5 (stale) Mirror 6: 0x2e35fc3f (stale) Mirror 7: 0x4af70b88 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xfe155ae5 (stale) Mirror 6: 0x2e35fc3f (stale) Mirror 7: 0x4af70b88 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000600 1787854490.189790 read 0x038cb8 thru 0x0390c6 (0x40f bytes) 000601 1787854490.200780 trunc from 0x040000 to 0x010c10 (0x2f3ef bytes) 000604 1787854490.225778 read 0x00b882 thru 0x010c10 (0x538f bytes) 000605 1787854490.234343 write 0x02b89a thru 0x0347c8 (0x8f2f bytes) 000607 1787854490.237295 trunc from 0x0347c9 to 0x00e30e (0x264ba bytes) 000608 1787854490.255737 read 0x00737d thru 0x00e30e (0x6f92 bytes) 000609 1787854490.256919 write 0x02dd8f thru 0x034f28 (0x719a bytes) 000610 1787854490.259317 write 0x0268ff thru 0x030c7a (0xa37c bytes) 000613 1787854490.269500 mapwrite 0x015377 thru 0x01729b (0x1f25 bytes) 000615 1787854490.317524 mapwrite 0x02182e thru 0x02ae0e (0x95e1 bytes) 000616 1787854490.364356 trunc from 0x034f29 to 0x022155 (0x12dd3 bytes) 000617 1787854490.392325 read 0x01b367 thru 0x022155 (0x6def bytes) 000618 1787854490.393423 read 0x00e5f2 thru 0x0103b1 (0x1dc0 bytes) 000622 1787854490.398032 mapwrite 0x015349 thru 0x015a15 (0x6cd bytes) 000623 1787854490.437230 write 0x00694e thru 0x008538 (0x1beb bytes) 000624 1787854490.438963 read 0x020940 thru 0x022155 (0x1816 bytes) 000625 1787854490.439760 trunc from 0x022156 to 0x002d33 (0x1f422 bytes) 000626 1787854490.464129 MIRROR_RESYNC 4 mirrors 000629 1787854491.477387 mapread 0x002a2c thru 0x002d33 (0x308 bytes) 000630 1787854491.488059 mapread 0x002af9 thru 0x002d33 (0x23b bytes) 000631 1787854491.490182 write 0x0016df thru 0x00cc74 (0xb596 bytes) 000632 1787854491.617977 mapread 0x009564 thru 0x00cc74 (0x3711 bytes) 000633 1787854491.626110 MIRROR_RESYNC 4 mirrors 000635 1787854492.715329 mapread 0x004a0b thru 0x00cc74 (0x826a bytes) 000636 1787854492.751300 write 0x03e402 thru 0x03ffff (0x1bfe bytes) 000638 1787854492.840582 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x76af9f8f (stale) Mirror 6: 0x76af9f8f (stale) Mirror 7: 0x76af9f8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x67bc5092 Mirror 8: 0x67bc5092 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000639 1787854494.179363 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x76af9f8f (stale) Mirror 6: 0x76af9f8f (stale) Mirror 7: 0x76af9f8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x67bc5092 Mirror 8: 0x67bc5092 Mirror 9: 0x67bc5092 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000640 1787854495.363154 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x76af9f8f (stale) Mirror 6: 0x76af9f8f (stale) Mirror 7: 0x76af9f8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x67bc5092 Mirror 8: 0x67bc5092 Mirror 9: 0x67bc5092 Mirror 10: 0x67bc5092 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000641 1787854496.516750 read 0x0124c9 thru 0x013f89 (0x1ac1 bytes) 000642 1787854496.533613 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x76af9f8f (stale) Mirror 7: 0x76af9f8f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x76af9f8f (stale) Mirror 7: 0x76af9f8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x67bc5092 Mirror 8: 0x67bc5092 Mirror 9: 0x67bc5092 Mirror 10: 0x67bc5092 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000643 1787854497.278901 mapread 0x01b4ce thru 0x020850 (0x5383 bytes) 000645 1787854497.332537 mapread 0x039f2a thru 0x03a844 (0x91b bytes) 000647 1787854497.352116 trunc from 0x040000 to 0x01d14b (0x22eb4 bytes) 000649 1787854497.553097 read 0x011215 thru 0x017aaa (0x6896 bytes) truncating to largest ever: 0x3fe3f 000650 1787854497.581595 trunc from 0x01d14c to 0x03fe3e (0x22cf3 bytes) 000651 1787854497.617791 trunc from 0x03fe3f to 0x0282a1 (0x17b9d bytes) 000652 1787854497.641085 trunc from 0x0282a2 to 0x028879 (0x5d8 bytes) 000653 1787854497.664891 trunc from 0x02887a to 0x032a0a (0xa191 bytes) 000654 1787854497.691200 MIRROR_RESYNC 6 mirrors 000656 1787854499.327419 write 0x0207fc thru 0x02bf67 (0xb76c bytes) 000658 1787854499.511402 mapread 0x0081ec thru 0x016adb (0xe8f0 bytes) 000659 1787854499.557261 write 0x034eb0 thru 0x03b090 (0x61e1 bytes) 000660 1787854499.559325 mapwrite 0x03de66 thru 0x03ffff (0x219a bytes) 000661 1787854499.646002 write 0x000734 thru 0x001fbc (0x1889 bytes) 000662 1787854499.663565 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x76af9f8f (stale) Mirror 7: 0x76af9f8f (stale) Mirror 8: 0x76af9f8f (stale) Mirror 9: 0x76af9f8f (stale) Mirror 10: 0x76af9f8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8ea636d Mirror 11: 0x8ea636d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000664 1787854500.956066 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x76af9f8f (stale) Mirror 7: 0x76af9f8f (stale) Mirror 8: 0x76af9f8f (stale) Mirror 9: 0x76af9f8f (stale) Mirror 10: 0x76af9f8f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8ea636d Mirror 11: 0x8ea636d Mirror 12: 0x8ea636d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000665 1787854502.163220 mapread 0x024c90 thru 0x0349bb (0xfd2c bytes) 000667 1787854502.219225 mapwrite 0x01bc2c thru 0x0211b4 (0x5589 bytes) 000668 1787854502.579371 trunc from 0x040000 to 0x0066df (0x39920 bytes) 000671 1787854502.611220 MIRROR_RESYNC 8 mirrors 000672 1787854504.612800 write 0x002854 thru 0x00ebc0 (0xc36d bytes) 000673 1787854504.717643 mapread 0x006908 thru 0x00bc17 (0x5310 bytes) 000676 1787854504.730262 write 0x02c2d3 thru 0x037783 (0xb4b1 bytes) 000677 1787854504.741364 trunc from 0x037784 to 0x00a918 (0x2ce6b bytes) 000680 1787854504.774541 mapread 0x00a1f9 thru 0x00a918 (0x720 bytes) 000681 1787854504.778813 trunc from 0x00a919 to 0x0111c3 (0x68ab bytes) 000682 1787854504.800043 trunc from 0x0111c4 to 0x03c57f (0x2b3bc bytes) 000683 1787854504.826625 mapread 0x01eb62 thru 0x02b170 (0xc60f bytes) 000684 1787854504.902430 MIRROR_RESYNC 8 mirrors 000685 1787854506.553237 MIRROR_RESYNC 8 mirrors 000686 1787854507.342290 trunc from 0x03c580 to 0x02a5d6 (0x11fa9 bytes) 000688 1787854507.457860 trunc from 0x02a5d7 to 0x039fb9 (0xf9e3 bytes) 000690 1787854507.478857 mapread 0x0362bb thru 0x039fb9 (0x3cff bytes) 000692 1787854507.509128 read 0x019b3e thru 0x021645 (0x7b08 bytes) 000696 1787854507.524817 write 0x02d0c6 thru 0x02f849 (0x2784 bytes) 000697 1787854507.539666 trunc from 0x039fba to 0x000432 (0x39b87 bytes) 000698 1787854507.564952 read 0x00030a thru 0x000432 (0x129 bytes) 000699 1787854507.570563 write 0x021d9e thru 0x029ea1 (0x8104 bytes) 000702 1787854507.578398 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x29ea2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29ea2): Mirror 5: 0x6c881c41 (stale) Mirror 7: 0x6c881c41 (stale) Mirror 8: 0x6c881c41 (stale) Mirror 9: 0x6c881c41 (stale) Mirror 10: 0x6c881c41 (stale) Mirror 11: 0x6c881c41 (stale) Mirror 12: 0x6c881c41 (stale) CRC-32 checksum value for chunk [0, 0x29ea2): Mirror 1: 0x57212175 Mirror 13: 0x57212175 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29ea2: skipped 000703 1787854509.184861 MIRROR_RESYNC 9 mirrors 000705 1787854511.198847 mapread 0x025c5a thru 0x029ea1 (0x4248 bytes) 000706 1787854511.232924 MIRROR_EXTEND to 10 mirrors 000707 1787854512.230121 read 0x01b696 thru 0x0233f3 (0x7d5e bytes) 000708 1787854512.238787 write 0x0013c8 thru 0x00671c (0x5355 bytes) 000709 1787854512.323012 read 0x00111d thru 0x008519 (0x73fd bytes) 000710 1787854512.338550 trunc from 0x029ea2 to 0x030661 (0x67c0 bytes) 000711 1787854512.371800 trunc from 0x030662 to 0x00d096 (0x235cb bytes) 000712 1787854512.391384 write 0x032747 thru 0x036d11 (0x45cb bytes) 000713 1787854512.396606 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x36d12): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d12): Mirror 5: 0xca9522d0 (stale) Mirror 7: 0xca9522d0 (stale) Mirror 8: 0xca9522d0 (stale) Mirror 9: 0xca9522d0 (stale) Mirror 10: 0xca9522d0 (stale) Mirror 11: 0xca9522d0 (stale) Mirror 12: 0xca9522d0 (stale) Mirror 13: 0xca9522d0 (stale) Mirror 14: 0xca9522d0 (stale) CRC-32 checksum value for chunk [0, 0x36d12): Mirror 1: 0xafa8ed68 Mirror 15: 0xafa8ed68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d12: skipped 000717 1787854514.043547 write 0x0193c8 thru 0x02187b (0x84b4 bytes) 000718 1787854514.137094 mapwrite 0x012d9b thru 0x0222b8 (0xf51e bytes) 000720 1787854514.218683 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d12): Mirror 7: 0xca9522d0 (stale) Mirror 8: 0xca9522d0 (stale) Mirror 9: 0xca9522d0 (stale) Mirror 10: 0xca9522d0 (stale) Mirror 11: 0xca9522d0 (stale) Mirror 12: 0xca9522d0 (stale) Mirror 13: 0xca9522d0 (stale) Mirror 14: 0xca9522d0 (stale) Mirror 15: 0xafa8ed68 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d12): Mirror 7: 0xca9522d0 (stale) Mirror 8: 0xca9522d0 (stale) Mirror 9: 0xca9522d0 (stale) Mirror 10: 0xca9522d0 (stale) Mirror 11: 0xca9522d0 (stale) Mirror 12: 0xca9522d0 (stale) Mirror 13: 0xca9522d0 (stale) Mirror 14: 0xca9522d0 (stale) Mirror 15: 0xafa8ed68 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d12: skipped 000721 1787854514.791672 read 0x036880 thru 0x036d11 (0x492 bytes) 000723 1787854514.798391 mapread 0x0155fb thru 0x01a825 (0x522b bytes) 000726 1787854514.817745 mapread 0x01b834 thru 0x021855 (0x6022 bytes) 000727 1787854514.826921 mapread 0x006b65 thru 0x015a64 (0xef00 bytes) 000728 1787854514.840333 mapwrite 0x01d3b5 thru 0x0256c0 (0x830c bytes) 000729 1787854514.890949 read 0x028c04 thru 0x02c095 (0x3492 bytes) 000730 1787854514.899974 read 0x0244f8 thru 0x026b48 (0x2651 bytes) 000731 1787854514.905060 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d12): Mirror 7: 0xca9522d0 (stale) Mirror 8: 0xca9522d0 (stale) Mirror 9: 0xca9522d0 (stale) Mirror 10: 0xca9522d0 (stale) Mirror 11: 0xca9522d0 (stale) Mirror 12: 0xca9522d0 (stale) Mirror 13: 0xca9522d0 (stale) Mirror 14: 0xca9522d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d12): Mirror 7: 0xca9522d0 (stale) Mirror 8: 0xca9522d0 (stale) Mirror 9: 0xca9522d0 (stale) Mirror 10: 0xca9522d0 (stale) Mirror 11: 0xca9522d0 (stale) Mirror 12: 0xca9522d0 (stale) Mirror 13: 0xca9522d0 (stale) Mirror 14: 0xca9522d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d12: skipped 000732 1787854515.342315 mapread 0x036095 thru 0x036d11 (0xc7d bytes) 000733 1787854515.350343 read 0x015218 thru 0x01d755 (0x853e bytes) 000734 1787854515.362211 mapread 0x0157f8 thru 0x01c19d (0x69a6 bytes) 000737 1787854515.364336 mapread 0x02836a thru 0x036d11 (0xe9a8 bytes) 000738 1787854515.386205 read 0x0097bf thru 0x0139de (0xa220 bytes) 000739 1787854515.395554 trunc from 0x036d12 to 0x027c54 (0xf0bd bytes) 000740 1787854515.410137 trunc from 0x027c55 to 0x03ae7b (0x13227 bytes) 000741 1787854515.430755 read 0x026f00 thru 0x027563 (0x664 bytes) 000742 1787854515.439902 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3ae7c): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae7c): Mirror 7: 0xf258825f (stale) Mirror 8: 0xf258825f (stale) Mirror 9: 0xf258825f (stale) Mirror 10: 0xf258825f (stale) Mirror 11: 0xf258825f (stale) Mirror 12: 0xf258825f (stale) Mirror 13: 0xf258825f (stale) Mirror 14: 0xf258825f (stale) CRC-32 checksum value for chunk [0, 0x3ae7c): Mirror 1: 0xf21388cb Mirror 15: 0xf21388cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae7c: skipped 000743 1787854516.814381 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ae7c): Mirror 7: 0xf258825f (stale) Mirror 8: 0xf258825f (stale) Mirror 9: 0xf258825f (stale) Mirror 10: 0xf258825f (stale) Mirror 11: 0xf258825f (stale) Mirror 12: 0xf258825f (stale) Mirror 13: 0xf258825f (stale) Mirror 14: 0xf258825f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ae7c): Mirror 7: 0xf258825f (stale) Mirror 8: 0xf258825f (stale) Mirror 9: 0xf258825f (stale) Mirror 10: 0xf258825f (stale) Mirror 11: 0xf258825f (stale) Mirror 12: 0xf258825f (stale) Mirror 13: 0xf258825f (stale) Mirror 14: 0xf258825f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ae7c: skipped 000745 1787854517.462514 write 0x032b05 thru 0x039869 (0x6d65 bytes) 000748 1787854517.485998 write 0x022930 thru 0x023187 (0x858 bytes) 000750 1787854517.502456 MIRROR_RESYNC 9 mirrors 000751 1787854519.522034 mapwrite 0x031184 thru 0x037bca (0x6a47 bytes) 000752 1787854519.789359 write 0x0391d1 thru 0x03ffff (0x6e2f bytes) 000753 1787854519.794492 mapread 0x026648 thru 0x0334db (0xce94 bytes) 000755 1787854519.812325 MIRROR_RESYNC 9 mirrors 000756 1787854521.435308 write 0x0198f6 thru 0x0298f4 (0xffff bytes) 000760 1787854521.542825 read 0x00d15c thru 0x00e20f (0x10b4 bytes) 000763 1787854521.554420 write 0x02dffe thru 0x0317ca (0x37cd bytes) 000766 1787854521.567592 mapwrite 0x0135f2 thru 0x022206 (0xec15 bytes) 000767 1787854521.687137 mapwrite 0x003e47 thru 0x006dc8 (0x2f82 bytes) 000769 1787854521.763685 trunc from 0x040000 to 0x031afd (0xe502 bytes) 000772 1787854521.792581 read 0x01a4b4 thru 0x01faba (0x5607 bytes) 000773 1787854521.804421 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31afe): Mirror 7: 0x12a55076 (stale) Mirror 8: 0x12a55076 (stale) Mirror 10: 0x12a55076 (stale) Mirror 11: 0x12a55076 (stale) Mirror 12: 0x12a55076 (stale) Mirror 13: 0x12a55076 (stale) Mirror 14: 0x12a55076 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31afe): Mirror 7: 0x12a55076 (stale) Mirror 8: 0x12a55076 (stale) Mirror 10: 0x12a55076 (stale) Mirror 11: 0x12a55076 (stale) Mirror 12: 0x12a55076 (stale) Mirror 13: 0x12a55076 (stale) Mirror 14: 0x12a55076 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31afe: skipped 000776 1787854522.364601 read 0x012d04 thru 0x014c96 (0x1f93 bytes) 000777 1787854522.370014 read 0x008ccb thru 0x01612a (0xd460 bytes) 000778 1787854522.389680 mapread 0x01a59e thru 0x027fa7 (0xda0a bytes) 000780 1787854522.392545 read 0x022afa thru 0x031afd (0xf004 bytes) 000784 1787854522.396261 mapread 0x00e4da thru 0x01e167 (0xfc8e bytes) 000785 1787854522.397846 read 0x03037a thru 0x031498 (0x111f bytes) 000788 1787854522.400402 mapread 0x022dd3 thru 0x02e259 (0xb487 bytes) 000790 1787854522.402851 write 0x03534f thru 0x03e1fc (0x8eae bytes) 000791 1787854522.404907 mapwrite 0x02d7fe thru 0x030d3e (0x3541 bytes) 000792 1787854522.457171 read 0x01ae56 thru 0x01fd72 (0x4f1d bytes) 000794 1787854522.460847 write 0x00c002 thru 0x018fc7 (0xcfc6 bytes) 000797 1787854522.471811 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 7: 0x1cc11558 (stale) Mirror 8: 0x1cc11558 (stale) Mirror 10: 0x1cc11558 (stale) Mirror 11: 0x1cc11558 (stale) Mirror 12: 0x1cc11558 (stale) Mirror 13: 0x1cc11558 (stale) Mirror 14: 0x1cc11558 (stale) CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 1: 0x8d3c7910 Mirror 15: 0x8d3c7910 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1fd: skipped 000798 1787854523.440518 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 7: 0x1cc11558 (stale) Mirror 8: 0x1cc11558 (stale) Mirror 10: 0x1cc11558 (stale) Mirror 11: 0x1cc11558 (stale) Mirror 12: 0x1cc11558 (stale) Mirror 14: 0x1cc11558 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 7: 0x1cc11558 (stale) Mirror 8: 0x1cc11558 (stale) Mirror 10: 0x1cc11558 (stale) Mirror 11: 0x1cc11558 (stale) Mirror 12: 0x1cc11558 (stale) Mirror 14: 0x1cc11558 (stale) CRC-32 checksum value for chunk [0, 0x3e1fd): Mirror 1: 0x8d3c7910 Mirror 15: 0x8d3c7910 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1fd: skipped 000799 1787854524.073904 mapwrite 0x007365 thru 0x00b24a (0x3ee6 bytes) 000800 1787854524.241059 mapwrite 0x008b8c thru 0x00cfe1 (0x4456 bytes) 000808 1787854524.297758 trunc from 0x03e1fd to 0x00e838 (0x2f9c4 bytes) 000812 1787854524.350035 mapwrite 0x00ec79 thru 0x01d675 (0xe9fd bytes) 000814 1787854524.426619 trunc from 0x01d676 to 0x00d829 (0xfe4c bytes) 000816 1787854524.448336 mapread 0x0054c9 thru 0x00d829 (0x8361 bytes) 000822 1787854524.453483 mapwrite 0x02017a thru 0x021e10 (0x1c97 bytes) 000824 1787854524.528283 read 0x0054a9 thru 0x008a02 (0x355a bytes) 000826 1787854524.533964 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x21e11): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21e11): Mirror 7: 0x3e0d4564 (stale) Mirror 8: 0x3e0d4564 (stale) Mirror 10: 0x3e0d4564 (stale) Mirror 11: 0x3e0d4564 (stale) Mirror 12: 0x3e0d4564 (stale) Mirror 14: 0x3e0d4564 (stale) Mirror 15: 0x947bf7e8 (stale) CRC-32 checksum value for chunk [0, 0x21e11): Mirror 1: 0xc4178d3c Mirror 16: 0xc4178d3c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21e11: skipped 000828 1787854525.662391 trunc from 0x021e11 to 0x03052b (0xe71b bytes) 000829 1787854525.770951 MIRROR_RESYNC 9 mirrors 000830 1787854528.669772 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) 000831 1787854529.255411 trunc from 0x03052c to 0x027cfa (0x8831 bytes) 000835 1787854529.358117 write 0x01ad88 thru 0x0248f4 (0x9b6d bytes) 000836 1787854529.374760 trunc from 0x027cfb to 0x01fb06 (0x81f4 bytes) 000838 1787854529.411522 read 0x00fadb thru 0x01d989 (0xdeaf bytes) 000841 1787854529.425255 read 0x0001bd thru 0x00fe03 (0xfc47 bytes) 000842 1787854529.438587 read 0x00c998 thru 0x01a449 (0xdab2 bytes) 000843 1787854529.439772 trunc from 0x01fb07 to 0x039b54 (0x1a04e bytes) 000844 1787854529.475656 trunc from 0x039b55 to 0x010c5a (0x28efa bytes) 000845 1787854529.499421 read 0x00dc40 thru 0x010c5a (0x301b bytes) 000846 1787854529.500108 trunc from 0x010c5b to 0x016232 (0x55d8 bytes) 000847 1787854529.559189 mapwrite 0x01b19e thru 0x0287f6 (0xd659 bytes) 000849 1787854529.700562 trunc from 0x0287f7 to 0x01a3a0 (0xe456 bytes) 000852 1787854529.731374 trunc from 0x01a3a1 to 0x01db4e (0x37ae bytes) 000853 1787854529.758293 mapwrite 0x0335ce thru 0x03607a (0x2aad bytes) 000854 1787854529.878718 read 0x02840a thru 0x034af0 (0xc6e7 bytes) 000857 1787854529.895795 trunc from 0x03607b to 0x0102dc (0x25d9e bytes) 000858 1787854529.931065 trunc from 0x0102dd to 0x01d29a (0xcfbe bytes) 000862 1787854529.961687 MIRROR_RESYNC 8 mirrors 000867 1787854531.523095 mapwrite 0x01b3b6 thru 0x0228b2 (0x74fd bytes) 000868 1787854531.902631 mapwrite 0x014933 thru 0x0241f7 (0xf8c5 bytes) 000869 1787854532.073028 mapread 0x0105ed thru 0x0126d4 (0x20e8 bytes) 000870 1787854532.074073 read 0x007896 thru 0x00949f (0x1c0a bytes) 000871 1787854532.091596 write 0x0344fa thru 0x03b5f2 (0x70f9 bytes) 000872 1787854532.101416 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3b5f3): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b5f3): Mirror 7: 0xf7902d36 (stale) Mirror 8: 0xf7902d36 (stale) Mirror 10: 0xf7902d36 (stale) Mirror 12: 0xf7902d36 (stale) Mirror 14: 0xf7902d36 (stale) Mirror 15: 0xf7902d36 (stale) Mirror 16: 0xf7902d36 (stale) CRC-32 checksum value for chunk [0, 0x3b5f3): Mirror 1: 0x66b3f32f Mirror 17: 0x66b3f32f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b5f3: skipped 000873 1787854533.407377 read 0x01c79b thru 0x025eb6 (0x971c bytes) 000874 1787854533.420265 write 0x0302a1 thru 0x0362ec (0x604c bytes) 000876 1787854533.535545 trunc from 0x03b5f3 to 0x03f99e (0x43ac bytes) 000878 1787854533.592368 trunc from 0x03f99f to 0x0381d0 (0x77ce bytes) 000879 1787854533.621218 mapread 0x0033f3 thru 0x00fcd6 (0xc8e4 bytes) 000880 1787854533.683853 read 0x018260 thru 0x022631 (0xa3d2 bytes) 000883 1787854533.700587 write 0x02c711 thru 0x03a4a0 (0xdd90 bytes) 000884 1787854533.703621 trunc from 0x03a4a1 to 0x004b9e (0x35902 bytes) 000885 1787854533.760277 MIRROR_RESYNC 9 mirrors 000887 1787854535.187276 trunc from 0x004b9f to 0x0274cf (0x22931 bytes) 000888 1787854535.358377 mapread 0x013f2a thru 0x01dee3 (0x9fba bytes) 000892 1787854535.409852 mapread 0x012312 thru 0x019498 (0x7187 bytes) 000894 1787854535.418885 write 0x01753f thru 0x019ab7 (0x2579 bytes) 000896 1787854535.421953 mapread 0x01cf67 thru 0x0274cf (0xa569 bytes) 000897 1787854535.458108 mapread 0x002e89 thru 0x0119a6 (0xeb1e bytes) 000898 1787854535.471485 mapwrite 0x00531e thru 0x00f335 (0xa018 bytes) 000900 1787854535.588625 MIRROR_RESYNC 9 mirrors 000902 1787854537.457304 read 0x012a01 thru 0x0203cc (0xd9cc bytes) 000903 1787854537.478240 MIRROR_EXTEND to 10 mirrors 000904 1787854538.799313 MIRROR_EXTEND to 11 mirrors 000906 1787854540.033588 read 0x024d40 thru 0x0274cf (0x2790 bytes) 000907 1787854540.043362 MIRROR_EXTEND to 12 mirrors 000909 1787854541.238576 read 0x006735 thru 0x00a3fc (0x3cc8 bytes) 000911 1787854541.248580 read 0x00efaa thru 0x01d39d (0xe3f4 bytes) 000912 1787854541.260408 trunc from 0x0274d0 to 0x03c236 (0x14d67 bytes) 000917 1787854541.419047 mapwrite 0x0303e0 thru 0x03216e (0x1d8f bytes) 000918 1787854541.549782 mapread 0x008bdf thru 0x0161e6 (0xd608 bytes) 000919 1787854541.656957 mapwrite 0x01bc8a thru 0x024b57 (0x8ece bytes) 000920 1787854541.763151 mapread 0x03ae2f thru 0x03c236 (0x1408 bytes) 000921 1787854541.775874 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c237): Mirror 7: 0xe674a74c (stale) Mirror 8: 0xe674a74c (stale) Mirror 10: 0xe674a74c (stale) Mirror 12: 0xe674a74c (stale) Mirror 14: 0xe674a74c (stale) Mirror 15: 0xe674a74c (stale) Mirror 17: 0xe674a74c (stale) Mirror 18: 0xe674a74c (stale) Mirror 19: 0xe674a74c (stale) Mirror 20: 0xe674a74c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c237): Mirror 7: 0xe674a74c (stale) Mirror 8: 0xe674a74c (stale) Mirror 10: 0xe674a74c (stale) Mirror 12: 0xe674a74c (stale) Mirror 14: 0xe674a74c (stale) Mirror 15: 0xe674a74c (stale) Mirror 17: 0xe674a74c (stale) Mirror 18: 0xe674a74c (stale) Mirror 19: 0xe674a74c (stale) Mirror 20: 0xe674a74c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c237: skipped 000922 1787854542.500537 write 0x000337 thru 0x000aec (0x7b6 bytes) 000923 1787854542.506789 write 0x014e32 thru 0x01ad62 (0x5f31 bytes) 000924 1787854542.517927 trunc from 0x03c237 to 0x003fd6 (0x38260 bytes) 000927 1787854542.554853 trunc from 0x003fd7 to 0x01b7e6 (0x17810 bytes) 000928 1787854542.574190 mapwrite 0x022b9f thru 0x02d4ea (0xa94c bytes) 000929 1787854542.679539 write 0x0327eb thru 0x03e4bd (0xbcd3 bytes) 000930 1787854542.699380 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3e4be): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4be): Mirror 7: 0x73f0e88e (stale) Mirror 8: 0x73f0e88e (stale) Mirror 10: 0x73f0e88e (stale) Mirror 12: 0x73f0e88e (stale) Mirror 14: 0x73f0e88e (stale) Mirror 15: 0x73f0e88e (stale) Mirror 17: 0x73f0e88e (stale) Mirror 18: 0x73f0e88e (stale) Mirror 19: 0x73f0e88e (stale) Mirror 20: 0x73f0e88e (stale) CRC-32 checksum value for chunk [0, 0x3e4be): Mirror 1: 0x24ab09f9 Mirror 21: 0x24ab09f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4be: skipped 000931 1787854544.105324 write 0x0307f3 thru 0x030f13 (0x721 bytes) 000933 1787854544.287380 mapread 0x0145f7 thru 0x01884a (0x4254 bytes) 000934 1787854544.317164 read 0x021eda thru 0x028a75 (0x6b9c bytes) 000936 1787854544.329340 trunc from 0x03e4be to 0x03dec8 (0x5f5 bytes) 000937 1787854544.362146 mapwrite 0x00d2de thru 0x017adb (0xa7fe bytes) 000938 1787854544.508188 write 0x0125cf thru 0x02174e (0xf180 bytes) 000940 1787854544.527930 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3dec9): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dec9): Mirror 7: 0x8c2b75fb (stale) Mirror 8: 0x8c2b75fb (stale) Mirror 10: 0x8c2b75fb (stale) Mirror 12: 0x8c2b75fb (stale) Mirror 14: 0x8c2b75fb (stale) Mirror 15: 0x8c2b75fb (stale) Mirror 17: 0x8c2b75fb (stale) Mirror 18: 0x8c2b75fb (stale) Mirror 19: 0x8c2b75fb (stale) Mirror 20: 0x8c2b75fb (stale) Mirror 21: 0x18fdbb37 (stale) CRC-32 checksum value for chunk [0, 0x3dec9): Mirror 1: 0xf4bc431d Mirror 22: 0xf4bc431d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dec9: skipped 000941 1787854545.684004 trunc from 0x03dec9 to 0x01d56e (0x2095a bytes) 000945 1787854545.809938 mapwrite 0x011398 thru 0x01fb00 (0xe769 bytes) 000946 1787854546.067333 MIRROR_SPLIT mirror 8 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fb01): Mirror 7: 0xb1eacbf1 (stale) Mirror 8: 0xb1eacbf1 (stale) Mirror 10: 0xb1eacbf1 (stale) Mirror 12: 0xb1eacbf1 (stale) Mirror 14: 0xb1eacbf1 (stale) Mirror 15: 0xb1eacbf1 (stale) Mirror 17: 0xb1eacbf1 (stale) Mirror 19: 0xb1eacbf1 (stale) Mirror 20: 0xb1eacbf1 (stale) Mirror 21: 0x3664cd77 (stale) Mirror 22: 0x4f53c77e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fb01): Mirror 7: 0xb1eacbf1 (stale) Mirror 8: 0xb1eacbf1 (stale) Mirror 10: 0xb1eacbf1 (stale) Mirror 12: 0xb1eacbf1 (stale) Mirror 14: 0xb1eacbf1 (stale) Mirror 15: 0xb1eacbf1 (stale) Mirror 17: 0xb1eacbf1 (stale) Mirror 19: 0xb1eacbf1 (stale) Mirror 20: 0xb1eacbf1 (stale) Mirror 21: 0x3664cd77 (stale) Mirror 22: 0x4f53c77e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fb01: skipped 000947 1787854546.872903 read 0x00956b thru 0x0182bb (0xed51 bytes) 000948 1787854546.886585 mapwrite 0x0065d5 thru 0x0067a3 (0x1cf bytes) 000952 1787854546.943348 mapwrite 0x03dd27 thru 0x03ffff (0x22d9 bytes) 000953 1787854547.014297 mapwrite 0x038ca9 thru 0x039538 (0x890 bytes) 000954 1787854547.055409 write 0x013cb2 thru 0x022d38 (0xf087 bytes) 000957 1787854547.063398 write 0x0163e7 thru 0x01eba2 (0x87bc bytes) 000959 1787854547.065115 trunc from 0x040000 to 0x0088a2 (0x3775d bytes) 000960 1787854547.082874 write 0x022fab thru 0x02c56e (0x95c4 bytes) 000962 1787854547.084778 mapwrite 0x00509a thru 0x013edb (0xee42 bytes) 000963 1787854547.195936 write 0x00e76d thru 0x0118a9 (0x313d bytes) 000965 1787854547.200254 mapread 0x00b7ca thru 0x019dad (0xe5e4 bytes) 000967 1787854547.203563 mapread 0x007161 thru 0x014031 (0xced1 bytes) 000968 1787854547.209967 read 0x009b1c thru 0x00eea5 (0x538a bytes) 000970 1787854547.211108 mapwrite 0x033e0c thru 0x03615a (0x234f bytes) 000971 1787854547.350343 mapwrite 0x03e47a thru 0x03ffff (0x1b86 bytes) 000972 1787854547.459831 mapread 0x029a6b thru 0x038aa7 (0xf03d bytes) 000976 1787854547.463633 read 0x030853 thru 0x036f25 (0x66d3 bytes) 000978 1787854547.465272 read 0x02656a thru 0x02661f (0xb6 bytes) 000979 1787854547.466586 write 0x03128e thru 0x03f02f (0xdda2 bytes) 000980 1787854547.468418 mapwrite 0x03c263 thru 0x03cf84 (0xd22 bytes) 000981 1787854547.555581 write 0x01fcd1 thru 0x0250a7 (0x53d7 bytes) 000983 1787854547.561311 read 0x00c5bc thru 0x00cfcc (0xa11 bytes) 000984 1787854547.569337 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x557f5790 (stale) Mirror 8: 0x557f5790 (stale) Mirror 12: 0x557f5790 (stale) Mirror 14: 0x557f5790 (stale) Mirror 15: 0x557f5790 (stale) Mirror 17: 0x557f5790 (stale) Mirror 19: 0x557f5790 (stale) Mirror 20: 0x557f5790 (stale) Mirror 21: 0x8b67c6e4 (stale) Mirror 22: 0xacc363a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x557f5790 (stale) Mirror 8: 0x557f5790 (stale) Mirror 12: 0x557f5790 (stale) Mirror 14: 0x557f5790 (stale) Mirror 15: 0x557f5790 (stale) Mirror 17: 0x557f5790 (stale) Mirror 19: 0x557f5790 (stale) Mirror 20: 0x557f5790 (stale) Mirror 21: 0x8b67c6e4 (stale) Mirror 22: 0xacc363a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000986 1787854548.294805 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x187:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x557f5790 (stale) Mirror 8: 0x557f5790 (stale) Mirror 14: 0x557f5790 (stale) Mirror 15: 0x557f5790 (stale) Mirror 17: 0x557f5790 (stale) Mirror 19: 0x557f5790 (stale) Mirror 20: 0x557f5790 (stale) Mirror 21: 0x8b67c6e4 (stale) Mirror 22: 0xacc363a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x557f5790 (stale) Mirror 8: 0x557f5790 (stale) Mirror 14: 0x557f5790 (stale) Mirror 15: 0x557f5790 (stale) Mirror 17: 0x557f5790 (stale) Mirror 19: 0x557f5790 (stale) Mirror 20: 0x557f5790 (stale) Mirror 21: 0x8b67c6e4 (stale) Mirror 22: 0xacc363a1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1787854548.926681 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x557f5790 (stale) Mirror 8: 0x557f5790 (stale) Mirror 14: 0x557f5790 (stale) Mirror 15: 0x557f5790 (stale) Mirror 17: 0x557f5790 (stale) Mirror 19: 0x557f5790 (stale) Mirror 20: 0x557f5790 (stale) Mirror 21: 0x8b67c6e4 (stale) Mirror 22: 0xacc363a1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x49e81bc1 Mirror 23: 0x49e81bc1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000989 1787854550.275809 trunc from 0x040000 to 0x02d3cc (0x12c33 bytes) 000990 1787854550.379376 read 0x001a41 thru 0x004b06 (0x30c6 bytes) 000992 1787854550.387435 mapread 0x00b3cb thru 0x0165bf (0xb1f5 bytes) 000993 1787854550.418876 trunc from 0x02d3cd to 0x03d35c (0xff90 bytes) 000994 1787854550.444377 MIRROR_RESYNC 11 mirrors 000996 1787854552.948633 read 0x034e04 thru 0x03767d (0x287a bytes) 000997 1787854552.961332 mapwrite 0x022976 thru 0x030c89 (0xe314 bytes) 000999 1787854553.204365 read 0x010cad thru 0x018dca (0x811e bytes) 001000 1787854553.260110 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3d35d): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d35d): Mirror 7: 0xa1a6693b (stale) Mirror 8: 0xa1a6693b (stale) Mirror 14: 0xa1a6693b (stale) Mirror 15: 0xa1a6693b (stale) Mirror 17: 0xa1a6693b (stale) Mirror 19: 0xa1a6693b (stale) Mirror 20: 0xa1a6693b (stale) Mirror 21: 0xa1a6693b (stale) Mirror 22: 0xa1a6693b (stale) Mirror 23: 0xa1a6693b (stale) CRC-32 checksum value for chunk [0, 0x3d35d): Mirror 1: 0xa537d49b Mirror 24: 0xa537d49b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d35d: skipped All operations completed A-OK! PASS 70a (113s) == sanity-flr test 71: check mirror extend parent fid ==== 14:15:59 (1787854559) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 14:16:02 (1787854562) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.349628 s, 6.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217595 s, 4.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 14:16:11 (1787854571) Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre2 Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7441600 .. Extending file size to 7122016 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7943808 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8431616 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8528000 .. Extending file size to 8736576 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8843392 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8605952 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58671 58672 58675 58676 58677 done 192.168.206.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg622-client.virtnet /mnt/lustre2 (opts:) 192.168.206.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg622-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:17:31 (1787854651) Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre2 Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8371328 .. Extending file size to 7254912 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7469952 .. failed Extending file size to 8527872 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8688864 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9162976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 60283 60284 60285 60289 60291 192.168.206.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg622-client.virtnet /mnt/lustre2 (opts:) 192.168.206.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg622-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:18:48 (1787854728) Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.175476 s, 12.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.316507 s, 6.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x57213adb (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.206.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg622-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:19:06 (1787854746) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124567 s, 16.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 14:19:14 (1787854754) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d7:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:19:22 (1787854762) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.287419 s, 7.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:19:30 (1787854770) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143148 s, 14.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:19:38 (1787854778) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.340317 s, 6.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:20:11 (1787854811) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg622-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126579 s, 8.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:21:03 (1787854863) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg622-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150149 s, 14.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:21:59 (1787854919) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg622-server mds-ost sync done. Creating new pool test_204f oleg622-server: Pool lustre.test_204f created Adding targets to pool oleg622-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg622-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131485 s, 15.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg622-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg622-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg622-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:23:17 (1787854997) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:23:24 (1787855004) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:23:31 (1787855011) oleg622-server: Pool lustre.test_206 created Adding targets to pool oleg622-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg622-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg622-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg622-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg622-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:23:55 (1787855035) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460104 s, 22.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246479 s, 42.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xec088a3f Mirror 3: 0xec088a3f PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:24:04 (1787855044) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:24:07 (1787855047) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:24:10 (1787855050) Starting client: oleg622-client.virtnet: -o user_xattr,flock 192.168.206.122@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.124249 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc70000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xcc0000 Read this (4 bytes): BBBB 192.168.206.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg622-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:24:51 (1787855091) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.029828 s, 275 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:24:58 (1787855098) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392101 s, 26.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fc lmm_fid: [0x200000404:0x1fc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fc lmm_fid: [0x200000404:0x1fc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834428 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 14:25:00.000000000 -0400 Modify: 2026-08-27 14:25:00.000000000 -0400 Change: 2026-08-27 14:25:01.000000000 -0400 Birth: 2026-08-27 14:24:59.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:25:07 (1787855107) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:25:10 (1787855110) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.559775 s, 73.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0864865 s, 47.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 14:25 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fe:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 14:25 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:25:20 (1787855120) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:25:28 (1787855128) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1798 sec =========== 14:25:36 (1787855136) === sanity-flr: start cleanup 14:25:38 (1787855138) === === sanity-flr: finish cleanup 14:25:45 (1787855145) ===