-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 12:09:42 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg402-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:10:07 (1787847007) === oleg402-client.virtnet: executing check_config_client /mnt/lustre oleg402-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg402-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8cd0a0413000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8cd0a0413000.idle_timeout=debug disable quota as required oleg402-server: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:10:48 (1787847048) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7634] == sanity-flr test 0a: lfs mirror create with -N option == 12:10:51 (1787847051) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:11:11 (1787847071) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:11:15 (1787847075) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:11:19 (1787847079) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:11:47 (1787847107) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:11:51 (1787847111) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 12:11:55 (1787847115) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8cd0a0413000.stats=clear osc.lustre-OST0001-osc-ffff8cd0a0413000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:12:05 (1787847125) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:12:19 (1787847139) PASS 0j (14s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:12:34 (1787847154) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:12:39 (1787847159) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (14s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:12:52 (1787847172) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 12:13:04 (1787847184) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:13:12 (1787847192) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:13:15 (1787847195) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:13:22 (1787847202) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:13:32 (1787847212) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:13:38 (1787847218) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.69203 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109913 s, 9.5 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:13:48 (1787847228) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.461017 s, 45.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:13:56 (1787847236) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:14:07 (1787847247) Stopping /mnt/lustre-ost1 (opts:) on oleg402-server oleg402-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cd0a0413000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd0a0413000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg402-server: oleg402-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg402-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg402-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg402-server oleg402-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cd0a0413000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd0a0413000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg402-server: oleg402-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg402-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (138s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:16:26 (1787847386) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.404502 s, 15.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg402-server oleg402-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cd0a0413000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd0a0413000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg402-server: oleg402-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg402-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:17:11 (1787847431) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg402-server oleg402-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cd0a0413000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd0a0413000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg402-server: oleg402-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg402-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg402-server oleg402-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cd0a0413000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd0a0413000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg402-server: oleg402-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg402-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:18:42 (1787847522) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.43811 s, 9.6 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 12:18 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.175s user 0m0.000s sys 0m0.202s Stopping /mnt/lustre-ost1 (opts:) on oleg402-server oleg402-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8cd0a0413000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8cd0a0413000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.245s user 0m0.001s sys 0m0.190s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg402-server: oleg402-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg402-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg402-server oleg402-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8cd0a0413000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd0a0413000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.274s user 0m0.001s sys 0m0.195s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg402-server: oleg402-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg402-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg402-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8cd0a0413000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8cd0a0413000.ost_server_uuid in FULL state after 0 sec PASS 33b (100s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:20:22 (1787847622) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:20:25 (1787847625) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:20:28 (1787847628) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 12:20:32 (1787847632) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 12:20:38 (1787847638) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.684181 s, 18.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:20:50 (1787847650) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047778 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331271 s, 31.7 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:20:56 (1787847656) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:21:03 (1787847663) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111449 s, 919 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00158041 s, 648 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00909242 s, 113 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00100296 s, 1.0 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 12:21:13 (1787847673) create test files with size 15751332 .. ea6dfb86fcbce372a979974fb2b289f2 - c5d2e128780f00e6198c9d56cdc771c1 - 23f41a323e349224f6067f7e7d75231f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 12:21:36 (1787847696) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:21:54 (1787847714) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:22:02 (1787847722) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180387 s, 5.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722362 s, 14.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 12:22:12 (1787847732) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.188929 s, 11.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.422703 s, 9.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.318502 s, 6.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 12:22:29 (1787847749) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 12:22:33 (1787847753) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:22:37 (1787847757) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310176 s, 33.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 12:22:45 (1787847765) llite.lustre-ffff8cd09827e000.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 12:22:54 (1787847774) llite.lustre-ffff8cd09827e000.enable_compression=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547091 s, 19.2 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422535 s, 24.8 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 12:23:03 (1787847783) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:23:07 (1787847787) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xda:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:23:20 (1787847800) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.333624 s, 31.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1f:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:23:36 (1787847816) llite.lustre-ffff8cd099309000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.306605 s, 34.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg402-client.virtnet /mnt/lustre (opts:) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8cd087110800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:23:52 (1787847832) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:24:35 (1787847875) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:25:13 (1787847913) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:25:17 (1787847917) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.352052 s, 8.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.258783 s, 12.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xea6d9a3d (nosync,stale) Mirror 2: 0xea6d9a3d (nosync,stale) ** verify mirror 2 doesn't change original checksum: f2bd9f7761ec2a314667afca57f510ee - mirror 1 checksum: 572995d1c6b7de7033ad62074f513fc4 - mirror 2 checksum: f2bd9f7761ec2a314667afca57f510ee - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T12:25:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T12:25:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f2bd9f7761ec2a314667afca57f510ee - mirror 1 checksum: 572995d1c6b7de7033ad62074f513fc4 - mirror 2 checksum: 572995d1c6b7de7033ad62074f513fc4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T12:25:20-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xef:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T12:25:20-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf0:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:25:31 (1787847931) Failing mds1 on oleg402-server Stopping /mnt/lustre-mds1 (opts:) on oleg402-server 12:25:38 (1787847938) shut down facet: mds1 facet_host: oleg402-server facet_failover_host: oleg402-server Failover mds1 to oleg402-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0000 12:26:04 (1787847964) targets are mounted 12:26:04 (1787847964) facet_failover done oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452758 s, 23.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xf1:0x0] } PASS 50A (51s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:26:23 (1787847983) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:26:27 (1787847987) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:26:31 (1787847991) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0136587 s, 1.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:26:40 (1787848000) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:26:44 (1787848004) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0152845 s, 268 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.114756 s, 35.7 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:26:53 (1787848013) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:27:18 (1787848038) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:27:47 (1787848067) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:28:11 (1787848091) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:28:37 (1787848117) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 12:28:45 (1787848125) mirror create pid 55598 mirror split pid 55600 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55598 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 55600 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 12:29:52 (1787848192) Seed set to 8194 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x12b4f 000001 1787848194.222082 trunc from 00000000 to 0x012b4e (0x12b4f bytes) 000002 1787848194.240502 mapwrite 0x03000a thru 0x035ad7 (0x5ace bytes) 000003 1787848194.396249 write 0x0043c3 thru 0x004d63 (0x9a1 bytes) 000004 1787848194.404533 trunc from 0x035ad8 to 0x00e312 (0x277c5 bytes) 000006 1787848194.429941 mapwrite 0x0276c3 thru 0x02954f (0x1e8d bytes) 000009 1787848194.509861 mapread 0x021f87 thru 0x02954f (0x75c9 bytes) 000012 1787848194.515113 mapwrite 0x004c9b thru 0x00a717 (0x5a7d bytes) 000014 1787848194.636216 read 0x027729 thru 0x02954f (0x1e27 bytes) 000015 1787848194.636846 read 0x01cc14 thru 0x025725 (0x8b12 bytes) 000016 1787848194.651012 mapread 0x0025b4 thru 0x003d59 (0x17a6 bytes) 000019 1787848194.661258 read 0x028368 thru 0x02954f (0x11e8 bytes) 000020 1787848194.662621 trunc from 0x029550 to 0x01258f (0x16fc0 bytes) truncating to largest ever: 0x3c66a 000021 1787848194.694774 trunc from 0x012590 to 0x03c669 (0x2a0da bytes) 000024 1787848194.724108 read 0x00611e thru 0x007d58 (0x1c3b bytes) 000026 1787848194.732266 read 0x035cb2 thru 0x03c669 (0x69b8 bytes) 000032 1787848194.761178 mapread 0x001638 thru 0x00518f (0x3b58 bytes) 000036 1787848194.763914 mapwrite 0x01905f thru 0x01b73f (0x26e1 bytes) 000040 1787848194.825944 trunc from 0x03c66a to 0x0081cb (0x3449e bytes) 000041 1787848194.856892 write 0x037dca thru 0x03ffff (0x8236 bytes) 000042 1787848194.859727 mapwrite 0x03c617 thru 0x03ffff (0x39e9 bytes) 000044 1787848194.916976 read 0x0298c2 thru 0x034e2a (0xb569 bytes) 000049 1787848194.938864 read 0x02d0d3 thru 0x031774 (0x46a2 bytes) 000050 1787848194.939580 mapread 0x00380e thru 0x01311c (0xf90f bytes) 000052 1787848194.942468 mapread 0x0288bb thru 0x02d4de (0x4c24 bytes) 000055 1787848194.945161 write 0x033316 thru 0x038e86 (0x5b71 bytes) 000061 1787848194.949802 read 0x0018e4 thru 0x00ec82 (0xd39f bytes) 000064 1787848194.951997 read 0x01c645 thru 0x026328 (0x9ce4 bytes) 000067 1787848194.954073 trunc from 0x040000 to 0x03a539 (0x5ac6 bytes) 000068 1787848194.970831 trunc from 0x03a53a to 0x008fea (0x3154f bytes) 000069 1787848194.996230 mapread 0x001625 thru 0x003feb (0x29c7 bytes) 000071 1787848194.997503 trunc from 0x008feb to 0x0006ed (0x88fd bytes) 000072 1787848195.027364 mapwrite 0x002610 thru 0x00fb41 (0xd532 bytes) 000077 1787848195.117849 mapread 0x008a90 thru 0x00fb41 (0x70b2 bytes) 000078 1787848195.119000 read 0x001e6a thru 0x007f89 (0x6120 bytes) 000079 1787848195.132693 MIRROR_EXTEND to 2 mirrors 000080 1787848195.990872 write 0x00b8cf thru 0x0184b4 (0xcbe6 bytes) 000081 1787848196.190357 trunc from 0x0184b5 to 0x001ce6 (0x167ce bytes) 000082 1787848196.246729 mapwrite 0x02d4b2 thru 0x038bdc (0xb72b bytes) 000083 1787848196.380357 mapwrite 0x039698 thru 0x03ffff (0x6968 bytes) 000092 1787848196.507754 read 0x00aa28 thru 0x011090 (0x6669 bytes) 000096 1787848196.526797 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdf0f8179 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4201bebd Mirror 3: 0x4201bebd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000097 1787848197.961809 MIRROR_RESYNC 3 mirrors 000099 1787848199.594921 mapread 0x005485 thru 0x014327 (0xeea3 bytes) 000100 1787848199.671749 read 0x01b211 thru 0x0241a1 (0x8f91 bytes) 000102 1787848199.683764 write 0x000ee7 thru 0x00c611 (0xb72b bytes) 000103 1787848199.811288 mapwrite 0x00ae47 thru 0x0176ca (0xc884 bytes) truncating to largest ever: 0x3fd90 000104 1787848200.076225 trunc from 0x040000 to 0x03fd8f (0x270 bytes) 000106 1787848200.124145 mapwrite 0x00be13 thru 0x0130c2 (0x72b0 bytes) 000107 1787848200.302843 mapread 0x029076 thru 0x02d332 (0x42bd bytes) 000108 1787848200.327419 trunc from 0x03fd90 to 0x03e98c (0x1403 bytes) 000114 1787848200.367146 read 0x03e3a6 thru 0x03e98c (0x5e7 bytes) 000115 1787848200.386659 trunc from 0x03e98d to 0x032cde (0xbcae bytes) 000116 1787848200.408484 mapwrite 0x009a4c thru 0x00e96c (0x4f21 bytes) 000118 1787848200.475664 trunc from 0x032cdf to 0x001744 (0x3159a bytes) 000119 1787848200.512464 mapread 0x0011b3 thru 0x001744 (0x592 bytes) 000120 1787848200.516599 trunc from 0x001745 to 0x01c572 (0x1ae2e bytes) 000122 1787848200.556702 mapwrite 0x03f571 thru 0x03ffff (0xa8f bytes) 000124 1787848200.729685 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4201bebd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4201bebd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000125 1787848201.276619 mapwrite 0x01e19a thru 0x023d32 (0x5b99 bytes) 000126 1787848201.467339 mapwrite 0x0242b8 thru 0x02c35e (0x80a7 bytes) 000127 1787848201.619977 mapwrite 0x0219fb thru 0x03068c (0xec92 bytes) 000128 1787848201.714506 MIRROR_RESYNC 2 mirrors 000129 1787848202.654735 trunc from 0x040000 to 0x03cfdb (0x3024 bytes) 000131 1787848202.760940 write 0x00319d thru 0x00c7a7 (0x960b bytes) 000134 1787848202.780053 write 0x0015d8 thru 0x0030e4 (0x1b0d bytes) 000136 1787848202.801762 trunc from 0x03cfdc to 0x016c02 (0x263d9 bytes) 000137 1787848202.844757 write 0x03801e thru 0x03fc1d (0x7c00 bytes) 000138 1787848202.847971 mapwrite 0x00c84d thru 0x00f21e (0x29d2 bytes) 000140 1787848202.957350 read 0x01521c thru 0x01a69e (0x5483 bytes) 000141 1787848202.971730 read 0x01c177 thru 0x024f25 (0x8daf bytes) 000142 1787848202.989028 mapread 0x02b766 thru 0x03038d (0x4c28 bytes) 000143 1787848202.990120 mapwrite 0x0184ac thru 0x02348d (0xafe2 bytes) 000144 1787848203.063419 mapread 0x00028f thru 0x003ffb (0x3d6d bytes) 000145 1787848203.069918 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000146 1787848203.226305 mapwrite 0x03b42d thru 0x03ffff (0x4bd3 bytes) 000147 1787848203.319650 trunc from 0x040000 to 0x01e341 (0x21cbe bytes) 000149 1787848203.335342 trunc from 0x01e342 to 0x0243dd (0x609c bytes) 000150 1787848203.351922 read 0x01b601 thru 0x01c0d1 (0xad1 bytes) 000151 1787848203.357249 write 0x031ccc thru 0x03d427 (0xb75c bytes) 000154 1787848203.360766 write 0x00a3ad thru 0x0104ac (0x6100 bytes) 000156 1787848203.370879 read 0x018f59 thru 0x023bef (0xac97 bytes) 000157 1787848203.384391 read 0x007ea7 thru 0x009305 (0x145f bytes) 000158 1787848203.385255 mapread 0x02f898 thru 0x03983e (0x9fa7 bytes) 000159 1787848203.386792 read 0x0218ec thru 0x028518 (0x6c2d bytes) 000162 1787848203.388717 mapread 0x034a87 thru 0x0393a1 (0x491b bytes) 000163 1787848203.389923 trunc from 0x03d428 to 0x010e07 (0x2c620 bytes) 000164 1787848203.404811 mapwrite 0x002cb1 thru 0x004bf5 (0x1f45 bytes) 000165 1787848203.458615 write 0x021e31 thru 0x031b21 (0xfcf1 bytes) 000166 1787848203.460875 read 0x005474 thru 0x008ab2 (0x363f bytes) 000168 1787848203.462803 mapwrite 0x02abcb thru 0x03662b (0xba61 bytes) 000169 1787848203.576177 mapwrite 0x002b56 thru 0x00dbe7 (0xb092 bytes) 000170 1787848203.663731 read 0x021704 thru 0x0286b1 (0x6fae bytes) 000171 1787848203.666183 trunc from 0x03662c to 0x00d3ca (0x29261 bytes) 000172 1787848203.686359 trunc from 0x00d3cb to 0x01ef42 (0x11b78 bytes) 000173 1787848203.701667 read 0x00bc9d thru 0x0113f0 (0x5754 bytes) 000177 1787848203.724672 mapread 0x00bc9a thru 0x00e2f6 (0x265d bytes) 000179 1787848203.726630 write 0x01e613 thru 0x024391 (0x5d7f bytes) 000180 1787848203.728790 read 0x01ec42 thru 0x023bdb (0x4f9a bytes) 000184 1787848203.734309 mapwrite 0x00908c thru 0x010edd (0x7e52 bytes) 000185 1787848203.798350 trunc from 0x024392 to 0x0003e1 (0x23fb0 bytes) 000186 1787848203.820984 read 0x000296 thru 0x0003e1 (0x14c bytes) 000190 1787848203.828286 read 0x00002c thru 0x0003e1 (0x3b6 bytes) 000193 1787848203.830724 read 0x000008 thru 0x0003e1 (0x3da bytes) 000194 1787848203.832589 mapwrite 0x030ea3 thru 0x032a30 (0x1b8e bytes) 000196 1787848203.913996 trunc from 0x032a31 to 0x014bfa (0x1de36 bytes) 000198 1787848203.961615 trunc from 0x014bfb to 0x00edb6 (0x5e44 bytes) 000203 1787848204.005765 mapwrite 0x023374 thru 0x02e7ad (0xb43a bytes) 000206 1787848204.375261 MIRROR_EXTEND to 2 mirrors 000208 1787848205.215633 mapread 0x00eb2b thru 0x013035 (0x450b bytes) 000209 1787848205.270187 mapread 0x012b3a thru 0x02134f (0xe816 bytes) 000210 1787848205.284156 write 0x00a299 thru 0x018521 (0xe289 bytes) 000211 1787848205.388099 read 0x02be4a thru 0x02e7ad (0x2964 bytes) 000212 1787848205.395919 mapread 0x018494 thru 0x020eb8 (0x8a25 bytes) 000214 1787848205.405755 write 0x007b66 thru 0x00848a (0x925 bytes) 000215 1787848205.415276 read 0x00b002 thru 0x00c1c8 (0x11c7 bytes) 000216 1787848205.415954 mapread 0x025b1a thru 0x028b3c (0x3023 bytes) 000218 1787848205.427345 mapread 0x02a779 thru 0x02e7ad (0x4035 bytes) 000219 1787848205.428741 write 0x013464 thru 0x01dcfa (0xa897 bytes) 000220 1787848205.434663 MIRROR_RESYNC 2 mirrors 000221 1787848206.867193 trunc from 0x02e7ae to 0x0341f3 (0x5a46 bytes) 000222 1787848207.009301 write 0x0209cd thru 0x02ae0e (0xa442 bytes) 000225 1787848207.051959 write 0x01edc1 thru 0x0245fd (0x583d bytes) 000227 1787848207.068921 trunc from 0x0341f4 to 0x03bc2f (0x7a3c bytes) 000230 1787848207.106331 mapwrite 0x01cfc8 thru 0x0282d6 (0xb30f bytes) 000232 1787848207.282102 mapwrite 0x03fb97 thru 0x03ffff (0x469 bytes) 000233 1787848207.438603 write 0x031222 thru 0x03bb20 (0xa8ff bytes) 000236 1787848207.449541 mapwrite 0x015615 thru 0x01a3e7 (0x4dd3 bytes) 000240 1787848207.563881 write 0x02b4b7 thru 0x032823 (0x736d bytes) 000241 1787848207.571821 mapread 0x014956 thru 0x022d67 (0xe412 bytes) 000243 1787848207.576441 trunc from 0x040000 to 0x01dea8 (0x22157 bytes) 000244 1787848207.612821 mapwrite 0x005a2f thru 0x010e8e (0xb460 bytes) 000245 1787848207.767524 trunc from 0x01dea9 to 0x00376e (0x1a73a bytes) 000250 1787848207.811293 trunc from 0x00376f to 0x01cc43 (0x194d5 bytes) 000253 1787848207.842829 mapwrite 0x0333cf thru 0x036478 (0x30aa bytes) 000254 1787848208.085090 write 0x008e4c thru 0x00fa26 (0x6bdb bytes) 000255 1787848208.113886 trunc from 0x036479 to 0x005076 (0x31402 bytes) 000256 1787848208.184129 mapread 0x00061d thru 0x005076 (0x4a5a bytes) 000257 1787848208.197011 trunc from 0x005077 to 0x03bee3 (0x36e6d bytes) 000258 1787848208.224863 trunc from 0x03bee4 to 0x03b9eb (0x4f8 bytes) 000260 1787848208.259021 MIRROR_RESYNC 2 mirrors 000261 1787848209.385038 mapwrite 0x02c33e thru 0x03afb1 (0xec74 bytes) 000262 1787848209.769496 mapwrite 0x01ea43 thru 0x02a4b7 (0xba75 bytes) 000265 1787848209.944226 read 0x03641e thru 0x03b9eb (0x55ce bytes) 000268 1787848209.949469 trunc from 0x03b9ec to 0x007587 (0x34464 bytes) 000269 1787848210.003037 read 0x001b47 thru 0x007587 (0x5a41 bytes) 000270 1787848210.026507 read 0x0057df thru 0x005c9a (0x4bc bytes) 000271 1787848210.028326 write 0x03bc99 thru 0x03ffff (0x4367 bytes) 000273 1787848210.036564 mapwrite 0x035e80 thru 0x03fba0 (0x9d21 bytes) 000274 1787848210.311113 read 0x016044 thru 0x01f85f (0x981c bytes) 000275 1787848210.312227 mapwrite 0x02d98e thru 0x035148 (0x77bb bytes) 000276 1787848210.369122 read 0x00e459 thru 0x00ea24 (0x5cc bytes) 000277 1787848210.369830 trunc from 0x040000 to 0x03ea9e (0x1561 bytes) 000278 1787848210.390594 trunc from 0x03ea9f to 0x029011 (0x15a8d bytes) 000279 1787848210.422629 read 0x01179d thru 0x01651d (0x4d81 bytes) 000284 1787848210.447897 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x29012): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29012): Mirror 2: 0x3dab5661 (stale) CRC-32 checksum value for chunk [0, 0x29012): Mirror 1: 0x3dab5661 Mirror 3: 0x3dab5661 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29012: skipped 000286 1787848211.879885 mapread 0x002184 thru 0x005926 (0x37a3 bytes) 000287 1787848211.929494 read 0x01c983 thru 0x026210 (0x988e bytes) 000289 1787848211.963231 read 0x008629 thru 0x00dce1 (0x56b9 bytes) 000290 1787848211.964821 mapwrite 0x00821b thru 0x00dcc9 (0x5aaf bytes) 000291 1787848212.193283 write 0x03b55e thru 0x03ffff (0x4aa2 bytes) 000295 1787848212.203624 MIRROR_RESYNC 3 mirrors 000297 1787848213.661111 write 0x0095e8 thru 0x00c02f (0x2a48 bytes) 000298 1787848213.759853 read 0x033ffb thru 0x039bad (0x5bb3 bytes) 000299 1787848213.778898 read 0x01a4e9 thru 0x024803 (0xa31b bytes) 000300 1787848213.797918 write 0x03740b thru 0x03ffff (0x8bf5 bytes) 000301 1787848213.799284 trunc from 0x040000 to 0x03ea84 (0x157b bytes) 000304 1787848213.833598 read 0x03bd86 thru 0x03ea84 (0x2cff bytes) 000305 1787848213.834584 mapread 0x03e981 thru 0x03ea84 (0x104 bytes) 000307 1787848213.836531 trunc from 0x03ea85 to 0x039687 (0x53fd bytes) 000308 1787848213.863855 write 0x035f05 thru 0x03ffff (0xa0fb bytes) 000311 1787848213.880052 MIRROR_RESYNC 3 mirrors 000312 1787848215.619696 write 0x003e69 thru 0x008f18 (0x50b0 bytes) 000313 1787848215.751087 trunc from 0x040000 to 0x02e518 (0x11ae7 bytes) 000314 1787848215.787646 mapwrite 0x0011d0 thru 0x00aa87 (0x98b8 bytes) 000315 1787848216.023758 trunc from 0x02e519 to 0x01afa3 (0x13575 bytes) 000316 1787848216.039778 mapread 0x013f47 thru 0x01afa3 (0x705d bytes) 000317 1787848216.064848 mapwrite 0x01f584 thru 0x020c89 (0x1706 bytes) 000319 1787848216.176921 trunc from 0x020c8a to 0x021eae (0x1225 bytes) 000320 1787848216.213349 write 0x0218ed thru 0x02e3e1 (0xcaf5 bytes) 000323 1787848216.223544 read 0x00262c thru 0x007d06 (0x56db bytes) 000324 1787848216.228280 mapwrite 0x02e467 thru 0x034457 (0x5ff1 bytes) 000325 1787848216.311534 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x34458): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34458): Mirror 2: 0x92e8f6f (stale) Mirror 3: 0x92e8f6f (stale) CRC-32 checksum value for chunk [0, 0x34458): Mirror 1: 0xe65359f7 Mirror 4: 0xe65359f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34458: skipped 000327 1787848217.797977 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34458): Mirror 3: 0x92e8f6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34458): Mirror 3: 0x92e8f6f (stale) CRC-32 checksum value for chunk [0, 0x34458): Mirror 1: 0xe65359f7 Mirror 4: 0xe65359f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34458: skipped 000329 1787848218.596883 read 0x012900 thru 0x01d244 (0xa945 bytes) 000330 1787848218.612225 trunc from 0x034458 to 0x000c7a (0x337dd bytes) 000331 1787848218.826137 trunc from 0x000c7b to 0x03105d (0x303e3 bytes) 000333 1787848218.882464 trunc from 0x03105e to 0x02196a (0xf6f3 bytes) 000336 1787848218.907865 mapwrite 0x014134 thru 0x01fa54 (0xb921 bytes) 000338 1787848219.053618 MIRROR_RESYNC 3 mirrors 000341 1787848220.349736 trunc from 0x02196b to 0x0190fc (0x886e bytes) 000342 1787848220.431448 read 0x000355 thru 0x002e83 (0x2b2f bytes) 000343 1787848220.440099 trunc from 0x0190fd to 0x03908a (0x1ff8e bytes) 000347 1787848220.470850 MIRROR_RESYNC 3 mirrors 000348 1787848221.754843 mapread 0x0323b7 thru 0x037924 (0x556e bytes) 000351 1787848221.793830 trunc from 0x03908b to 0x001909 (0x37781 bytes) 000352 1787848221.880433 trunc from 0x00190a to 0x031973 (0x3006a bytes) 000354 1787848221.903641 write 0x03a628 thru 0x03ffff (0x59d8 bytes) 000355 1787848221.907023 mapread 0x030943 thru 0x03a583 (0x9c41 bytes) 000357 1787848221.950685 mapwrite 0x034612 thru 0x03ffff (0xb9ee bytes) 000358 1787848222.062569 write 0x01e8d2 thru 0x02d640 (0xed6f bytes) 000359 1787848222.073482 write 0x032374 thru 0x037549 (0x51d6 bytes) 000362 1787848222.088494 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9a6f21fd (stale) Mirror 4: 0x9a6f21fd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xccd78836 Mirror 5: 0xccd78836 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000365 1787848223.646897 mapread 0x019620 thru 0x019b6d (0x54e bytes) 000368 1787848223.654911 read 0x02a6a6 thru 0x038aec (0xe447 bytes) 000371 1787848223.668043 read 0x0052cf thru 0x00f779 (0xa4ab bytes) 000373 1787848223.711823 MIRROR_RESYNC 4 mirrors 000374 1787848224.964266 write 0x020f04 thru 0x026583 (0x5680 bytes) 000375 1787848225.050555 mapread 0x013995 thru 0x01fd45 (0xc3b1 bytes) 000376 1787848225.082934 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xccd78836 (stale) Mirror 4: 0xccd78836 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xccd78836 (stale) Mirror 4: 0xccd78836 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000377 1787848225.726166 mapread 0x00a4ab thru 0x011fa7 (0x7afd bytes) 000378 1787848225.758845 trunc from 0x040000 to 0x0267cd (0x19832 bytes) 000379 1787848225.781519 mapread 0x01ffd9 thru 0x0267cd (0x67f5 bytes) 000380 1787848225.797568 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x267ce): Mirror 4: 0x76592faf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x267ce): Mirror 4: 0x76592faf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x267ce: skipped 000381 1787848226.323772 trunc from 0x0267ce to 0x02b4c5 (0x4cf8 bytes) 000382 1787848226.354016 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000383 1787848226.527149 write 0x01c691 thru 0x0240db (0x7a4b bytes) 000384 1787848226.543775 mapread 0x012dd7 thru 0x01baf0 (0x8d1a bytes) 000388 1787848226.574372 mapread 0x0068ae thru 0x0082e1 (0x1a34 bytes) 000389 1787848226.585817 trunc from 0x02b4c6 to 0x038ef4 (0xda2f bytes) 000393 1787848226.626199 MIRROR_EXTEND to 2 mirrors 000395 1787848227.781016 write 0x03eb3a thru 0x03ffff (0x14c6 bytes) 000397 1787848227.892254 mapwrite 0x03f235 thru 0x03ffff (0xdcb bytes) 000400 1787848227.954299 mapread 0x01713a thru 0x0184b5 (0x137c bytes) 000401 1787848227.966771 read 0x012017 thru 0x01c3b5 (0xa39f bytes) 000403 1787848227.982512 mapwrite 0x021b36 thru 0x02282f (0xcfa bytes) 000404 1787848228.044501 trunc from 0x040000 to 0x024eb8 (0x1b147 bytes) 000405 1787848228.076451 write 0x03c97a thru 0x03ffff (0x3686 bytes) 000409 1787848228.081001 read 0x02f629 thru 0x035509 (0x5ee1 bytes) 000410 1787848228.093118 mapwrite 0x016cc0 thru 0x01e1bb (0x74fc bytes) 000411 1787848228.152016 mapread 0x018e04 thru 0x024bb3 (0xbdb0 bytes) 000412 1787848228.154960 read 0x0252db thru 0x02d0b6 (0x7ddc bytes) 000413 1787848228.156006 mapread 0x01c3b9 thru 0x01c3e9 (0x31 bytes) 000417 1787848228.159737 write 0x00aa1a thru 0x016d00 (0xc2e7 bytes) 000419 1787848228.162258 trunc from 0x040000 to 0x002dc0 (0x3d23f bytes) 000421 1787848228.183757 read 0x000e90 thru 0x002dc0 (0x1f31 bytes) 000422 1787848228.184768 write 0x036745 thru 0x03ffff (0x98bb bytes) 000424 1787848228.188560 read 0x01a1bc thru 0x02149c (0x72e1 bytes) 000425 1787848228.198530 mapwrite 0x0088ac thru 0x0152bd (0xca12 bytes) 000427 1787848228.261564 mapread 0x02f7fc thru 0x03c0bd (0xc8c2 bytes) 000429 1787848228.269765 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000430 1787848228.482107 write 0x017a98 thru 0x01a459 (0x29c2 bytes) 000431 1787848228.496242 trunc from 0x040000 to 0x037ae0 (0x851f bytes) 000434 1787848228.514867 trunc from 0x037ae1 to 0x01bf6d (0x1bb73 bytes) 000435 1787848228.533094 trunc from 0x01bf6e to 0x0267ce (0xa861 bytes) 000436 1787848228.551303 read 0x01dd72 thru 0x0267ce (0x8a5d bytes) 000437 1787848228.566157 read 0x020f7d thru 0x0267ce (0x5852 bytes) 000439 1787848228.567469 read 0x020d04 thru 0x022d58 (0x2055 bytes) 000441 1787848228.569338 write 0x019782 thru 0x025578 (0xbdf7 bytes) 000444 1787848228.574270 mapread 0x00ee65 thru 0x019a20 (0xabbc bytes) 000445 1787848228.586966 read 0x00ea89 thru 0x015880 (0x6df8 bytes) 000446 1787848228.588087 mapread 0x0125ce thru 0x01a200 (0x7c33 bytes) 000449 1787848228.591926 write 0x017315 thru 0x019c00 (0x28ec bytes) 000450 1787848228.593334 mapwrite 0x0364cb thru 0x03ffff (0x9b35 bytes) 000453 1787848228.695943 read 0x03b86b thru 0x03c9dd (0x1173 bytes) 000454 1787848228.697113 mapread 0x01865e thru 0x026e37 (0xe7da bytes) 000455 1787848228.699092 write 0x019894 thru 0x01e2a8 (0x4a15 bytes) 000460 1787848228.705162 trunc from 0x040000 to 0x0320a0 (0xdf5f bytes) 000461 1787848228.729904 read 0x0060d8 thru 0x011922 (0xb84b bytes) 000462 1787848228.732976 mapwrite 0x0193eb thru 0x01e0e1 (0x4cf7 bytes) 000463 1787848228.808104 mapwrite 0x01f6db thru 0x026b05 (0x742b bytes) 000464 1787848228.861054 mapwrite 0x01bd45 thru 0x0225a2 (0x685e bytes) 000466 1787848229.000108 mapread 0x030bb9 thru 0x0320a0 (0x14e8 bytes) 000468 1787848229.007469 mapwrite 0x026750 thru 0x0288e9 (0x219a bytes) 000469 1787848229.067025 mapread 0x02e777 thru 0x0320a0 (0x392a bytes) 000470 1787848229.068506 mapread 0x0261fe thru 0x02c291 (0x6094 bytes) 000471 1787848229.070271 read 0x017f5b thru 0x0206f8 (0x879e bytes) 000474 1787848229.073005 mapwrite 0x001049 thru 0x004e16 (0x3dce bytes) 000475 1787848229.184710 mapwrite 0x002f3e thru 0x00c87e (0x9941 bytes) 000478 1787848229.244429 write 0x007392 thru 0x014e14 (0xda83 bytes) 000481 1787848229.248726 mapwrite 0x03439a thru 0x037897 (0x34fe bytes) 000482 1787848229.318257 mapread 0x02d090 thru 0x02dc59 (0xbca bytes) 000483 1787848229.319296 write 0x00b71a thru 0x01409b (0x8982 bytes) 000486 1787848229.323128 write 0x02c3e1 thru 0x030813 (0x4433 bytes) 000487 1787848229.324618 mapwrite 0x032c15 thru 0x037598 (0x4984 bytes) 000491 1787848229.390577 mapread 0x026c8e thru 0x02b7fd (0x4b70 bytes) 000492 1787848229.394827 read 0x022ba9 thru 0x025f6b (0x33c3 bytes) 000493 1787848229.400533 trunc from 0x037898 to 0x015ba8 (0x21cef bytes) 000494 1787848229.426241 mapwrite 0x014f7a thru 0x015393 (0x41a bytes) 000495 1787848229.515850 trunc from 0x015ba9 to 0x01c9ed (0x6e45 bytes) 000496 1787848229.562306 mapwrite 0x004cfd thru 0x00b758 (0x6a5c bytes) 000498 1787848229.609747 mapwrite 0x035c0d thru 0x037345 (0x1739 bytes) 000500 1787848229.680761 trunc from 0x037346 to 0x02e47d (0x8ec8 bytes) 000501 1787848229.703705 trunc from 0x02e47e to 0x0239c8 (0xaab5 bytes) 000503 1787848229.731487 mapread 0x01e317 thru 0x0239c8 (0x56b2 bytes) 000504 1787848229.733774 read 0x00b060 thru 0x018fd3 (0xdf74 bytes) 000505 1787848229.735516 trunc from 0x0239c9 to 0x035c49 (0x12281 bytes) 000507 1787848229.786144 mapwrite 0x018e36 thru 0x02013c (0x7307 bytes) 000508 1787848229.868693 trunc from 0x035c4a to 0x024277 (0x119d2 bytes) 000510 1787848229.897785 mapread 0x014b86 thru 0x022591 (0xda0c bytes) 000511 1787848229.901709 read 0x007d8a thru 0x008d8e (0x1005 bytes) 000512 1787848229.902619 read 0x018bdb thru 0x024277 (0xb69d bytes) 000513 1787848229.911556 trunc from 0x024278 to 0x02ccc7 (0x8a50 bytes) 000517 1787848229.953953 mapread 0x0104d1 thru 0x01eaeb (0xe61b bytes) 000520 1787848229.956825 read 0x00cc6d thru 0x010e8e (0x4222 bytes) 000521 1787848229.958161 trunc from 0x02ccc8 to 0x01f5bf (0xd708 bytes) 000524 1787848229.989202 trunc from 0x01f5c0 to 0x0389df (0x19420 bytes) 000525 1787848230.017905 write 0x02007d thru 0x028dad (0x8d31 bytes) 000526 1787848230.037195 mapwrite 0x011271 thru 0x011a4f (0x7df bytes) 000527 1787848230.110472 read 0x0240e6 thru 0x02c30b (0x8226 bytes) 000529 1787848230.114044 mapread 0x005ffc thru 0x013b0c (0xdb11 bytes) 000530 1787848230.118561 mapwrite 0x0228bb thru 0x023fae (0x16f4 bytes) 000531 1787848230.185903 mapread 0x014796 thru 0x014a91 (0x2fc bytes) 000532 1787848230.186819 write 0x0119fb thru 0x011f59 (0x55f bytes) 000533 1787848230.187737 write 0x02e42f thru 0x035465 (0x7037 bytes) 000534 1787848230.191296 mapwrite 0x03fa59 thru 0x03ffff (0x5a7 bytes) 000536 1787848230.284716 mapread 0x01d847 thru 0x024bfc (0x73b6 bytes) 000537 1787848230.287263 trunc from 0x040000 to 0x014630 (0x2b9cf bytes) 000541 1787848230.336266 read 0x00aaba thru 0x00b3bd (0x904 bytes) 000542 1787848230.341297 trunc from 0x014631 to 0x0235cc (0xef9c bytes) 000543 1787848230.387454 trunc from 0x0235cd to 0x03457e (0x10fb2 bytes) 000546 1787848230.431607 MIRROR_EXTEND to 2 mirrors 000549 1787848231.872317 trunc from 0x03457f to 0x00efd1 (0x255ad bytes) 000552 1787848232.108510 read 0x000de8 thru 0x00c745 (0xb95e bytes) 000553 1787848232.145434 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xefd2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xefd2): Mirror 2: 0x1be9320d (stale) CRC-32 checksum value for chunk [0, 0xefd2): Mirror 1: 0x1be9320d Mirror 3: 0x1be9320d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xefd2: skipped 000554 1787848233.663956 read 0x002cb5 thru 0x0064f2 (0x383e bytes) 000555 1787848233.679051 write 0x03e0f9 thru 0x03ffff (0x1f07 bytes) 000556 1787848233.775775 mapwrite 0x01ed0e thru 0x02b41f (0xc712 bytes) 000560 1787848233.933804 mapread 0x00f486 thru 0x01ebf7 (0xf772 bytes) 000561 1787848233.971456 read 0x037c6c thru 0x03ffff (0x8394 bytes) 000562 1787848234.011259 mapread 0x00430d thru 0x00e5cd (0xa2c1 bytes) 000566 1787848234.020048 mapwrite 0x03ac5c thru 0x03bede (0x1283 bytes) 000568 1787848234.094782 write 0x02263e thru 0x025612 (0x2fd5 bytes) 000570 1787848234.098755 mapwrite 0x0116e5 thru 0x01ed63 (0xd67f bytes) 000574 1787848234.165988 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e73d2cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2e73d2cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000576 1787848234.581170 mapwrite 0x0265c8 thru 0x035a23 (0xf45c bytes) 000578 1787848234.748575 write 0x023ae2 thru 0x023eaf (0x3ce bytes) 000579 1787848234.749568 trunc from 0x040000 to 0x008451 (0x37bae bytes) 000582 1787848234.834100 trunc from 0x008452 to 0x023016 (0x1abc5 bytes) 000585 1787848234.882800 trunc from 0x023017 to 0x0011f5 (0x21e21 bytes) 000586 1787848234.919760 trunc from 0x0011f6 to 0x0146f6 (0x13501 bytes) 000587 1787848234.974026 mapwrite 0x00c06f thru 0x01b38a (0xf31c bytes) 000588 1787848235.188690 write 0x01f1a0 thru 0x0227d7 (0x3638 bytes) 000589 1787848235.193231 trunc from 0x0227d8 to 0x017383 (0xb454 bytes) 000590 1787848235.212159 mapread 0x00ac26 thru 0x017383 (0xc75e bytes) 000591 1787848235.213465 read 0x000d98 thru 0x00287d (0x1ae6 bytes) 000592 1787848235.222755 mapread 0x008cf7 thru 0x00da81 (0x4d8b bytes) 000593 1787848235.224151 write 0x000f92 thru 0x00d496 (0xc505 bytes) 000595 1787848235.228061 read 0x009ab3 thru 0x00b451 (0x199f bytes) 000596 1787848235.229301 write 0x014896 thru 0x01dc5b (0x93c6 bytes) 000597 1787848235.231756 mapwrite 0x0018a4 thru 0x0022b0 (0xa0d bytes) 000600 1787848235.296280 write 0x037eba thru 0x03ffff (0x8146 bytes) 000601 1787848235.301750 trunc from 0x040000 to 0x00c3f5 (0x33c0a bytes) 000603 1787848235.328870 mapread 0x00321a thru 0x00818b (0x4f72 bytes) 000604 1787848235.331408 write 0x02d136 thru 0x02dfe3 (0xeae bytes) 000605 1787848235.333900 read 0x0220e6 thru 0x02dfe3 (0xbefe bytes) 000608 1787848235.344000 mapread 0x02988d thru 0x02b8cc (0x2040 bytes) 000609 1787848235.349994 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000610 1787848235.512091 mapread 0x019652 thru 0x023110 (0x9abf bytes) 000611 1787848235.545598 mapwrite 0x00304b thru 0x00ccf3 (0x9ca9 bytes) 000615 1787848235.639661 write 0x0215ea thru 0x025c44 (0x465b bytes) 000616 1787848235.649967 mapwrite 0x0290a7 thru 0x02c225 (0x317f bytes) 000618 1787848235.756965 mapread 0x004e6f thru 0x0116fe (0xc890 bytes) 000619 1787848235.769796 MIRROR_EXTEND to 2 mirrors 000620 1787848236.970866 write 0x037a4a thru 0x037a9e (0x55 bytes) 000621 1787848237.121077 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37a9f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a9f): Mirror 2: 0x2c6541c8 (stale) CRC-32 checksum value for chunk [0, 0x37a9f): Mirror 1: 0x674a6fb1 Mirror 3: 0x674a6fb1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a9f: skipped 000622 1787848238.485653 mapwrite 0x03d048 thru 0x03ffff (0x2fb8 bytes) 000623 1787848238.641036 mapwrite 0x03c252 thru 0x03e92e (0x26dd bytes) 000624 1787848238.722067 MIRROR_RESYNC 3 mirrors 000625 1787848240.351831 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000626 1787848240.685606 write 0x023814 thru 0x02f800 (0xbfed bytes) 000630 1787848240.842847 mapread 0x030ad5 thru 0x033719 (0x2c45 bytes) 000631 1787848240.896736 MIRROR_RESYNC 2 mirrors 000632 1787848242.282687 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000635 1787848242.431971 MIRROR_EXTEND to 2 mirrors 000637 1787848243.598615 mapread 0x014f7a thru 0x015cfe (0xd85 bytes) 000640 1787848243.625665 write 0x012e19 thru 0x01b345 (0x852d bytes) 000641 1787848243.702817 read 0x016c68 thru 0x019958 (0x2cf1 bytes) 000644 1787848243.704351 trunc from 0x040000 to 0x025470 (0x1ab8f bytes) 000645 1787848243.736684 mapread 0x020cb5 thru 0x025470 (0x47bc bytes) 000646 1787848243.759201 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x25471): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25471): Mirror 2: 0x6536c146 (stale) CRC-32 checksum value for chunk [0, 0x25471): Mirror 1: 0x5aed131e Mirror 3: 0x5aed131e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25471: skipped 000647 1787848244.841766 read 0x0250eb thru 0x025470 (0x386 bytes) 000649 1787848244.848756 read 0x01a89d thru 0x021ceb (0x744f bytes) 000650 1787848244.866952 write 0x02970f thru 0x029b4d (0x43f bytes) 000652 1787848244.970193 MIRROR_RESYNC 3 mirrors 000653 1787848245.979998 write 0x03d30c thru 0x03ffff (0x2cf4 bytes) 000656 1787848246.103565 MIRROR_RESYNC 3 mirrors 000657 1787848247.333559 write 0x01029a thru 0x0194ea (0x9251 bytes) 000659 1787848247.542543 MIRROR_RESYNC 3 mirrors 000660 1787848248.746773 write 0x00fff8 thru 0x011561 (0x156a bytes) 000661 1787848248.852854 read 0x032d2f thru 0x03d3f4 (0xa6c6 bytes) 000662 1787848248.867206 trunc from 0x040000 to 0x00a239 (0x35dc6 bytes) 000665 1787848248.895481 mapwrite 0x02c0be thru 0x02ccdb (0xc1e bytes) 000666 1787848248.968924 write 0x02a28f thru 0x036928 (0xc69a bytes) 000669 1787848248.976611 trunc from 0x036929 to 0x02d9e0 (0x8f48 bytes) 000671 1787848248.998360 trunc from 0x02d9e1 to 0x01d9e8 (0xfff8 bytes) 000675 1787848249.018858 mapwrite 0x03395a thru 0x03cc76 (0x931d bytes) 000676 1787848249.186591 mapread 0x03c51e thru 0x03cc76 (0x759 bytes) 000677 1787848249.188063 read 0x03c1c8 thru 0x03cc76 (0xaaf bytes) 000678 1787848249.189615 read 0x0324eb thru 0x0325ed (0x103 bytes) 000679 1787848249.190571 mapwrite 0x015726 thru 0x01901e (0x38f9 bytes) 000680 1787848249.268196 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cc77): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc77): Mirror 2: 0x24e87833 (stale) Mirror 3: 0x24e87833 (stale) CRC-32 checksum value for chunk [0, 0x3cc77): Mirror 1: 0xaca449c6 Mirror 4: 0xaca449c6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc77: skipped 000682 1787848250.763830 read 0x026904 thru 0x032c69 (0xc366 bytes) 000683 1787848250.776635 trunc from 0x03cc77 to 0x01887c (0x243fa bytes) 000685 1787848250.880854 mapread 0x005e1e thru 0x011c85 (0xbe68 bytes) 000686 1787848250.923167 mapwrite 0x039160 thru 0x039919 (0x7ba bytes) 000687 1787848251.029478 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3991a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3991a): Mirror 2: 0x6e118e05 (stale) Mirror 3: 0x6e118e05 (stale) Mirror 4: 0xe77bfc8b (stale) CRC-32 checksum value for chunk [0, 0x3991a): Mirror 1: 0xf06f1a5d Mirror 5: 0xf06f1a5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3991a: skipped 000688 1787848252.684034 read 0x028c4c thru 0x034a67 (0xbe1c bytes) 000690 1787848252.703355 read 0x025bba thru 0x02edd0 (0x9217 bytes) 000691 1787848252.719295 mapwrite 0x019fbd thru 0x01e54b (0x458f bytes) 000692 1787848252.973845 read 0x000e62 thru 0x00d2fe (0xc49d bytes) 000695 1787848253.026909 MIRROR_RESYNC 5 mirrors 000698 1787848254.995949 trunc from 0x03991a to 0x005e34 (0x33ae5 bytes) 000700 1787848255.112229 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x5e35): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5e35): Mirror 2: 0xd7833f1a (stale) Mirror 3: 0xd7833f1a (stale) Mirror 4: 0xd7833f1a (stale) Mirror 5: 0xd7833f1a (stale) CRC-32 checksum value for chunk [0, 0x5e35): Mirror 1: 0xd7833f1a Mirror 6: 0xd7833f1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5e35: skipped 000702 1787848256.275542 mapwrite 0x0086bf thru 0x0146fc (0xc03e bytes) 000703 1787848256.452871 trunc from 0x0146fd to 0x03a111 (0x25a15 bytes) 000704 1787848256.488919 mapread 0x022898 thru 0x02ddd7 (0xb540 bytes) 000705 1787848256.513892 write 0x009e5a thru 0x015808 (0xb9af bytes) 000706 1787848256.520411 write 0x03331a thru 0x03a3da (0x70c1 bytes) 000707 1787848256.533543 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a3db): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3db): Mirror 2: 0xda45b47d (stale) Mirror 3: 0xda45b47d (stale) Mirror 4: 0xda45b47d (stale) Mirror 5: 0xda45b47d (stale) Mirror 6: 0x95a6ce4e (stale) CRC-32 checksum value for chunk [0, 0x3a3db): Mirror 1: 0xbbbc88d8 Mirror 7: 0xbbbc88d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3db: skipped 000715 1787848257.891426 trunc from 0x03a3db to 0x01b48b (0x1ef4f bytes) 000716 1787848258.073344 mapwrite 0x006af5 thru 0x0160f2 (0xf5fe bytes) 000717 1787848258.274665 read 0x0108bd thru 0x012a89 (0x21cd bytes) 000719 1787848258.275813 mapread 0x00e1f4 thru 0x014ca0 (0x6aad bytes) 000722 1787848258.282846 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1b48c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b48c): Mirror 2: 0x3cd9b3de (stale) Mirror 3: 0x3cd9b3de (stale) Mirror 4: 0x3cd9b3de (stale) Mirror 5: 0x3cd9b3de (stale) Mirror 6: 0xfee04e61 (stale) Mirror 7: 0x9d26f051 (stale) CRC-32 checksum value for chunk [0, 0x1b48c): Mirror 1: 0x47cb7bb6 Mirror 8: 0x47cb7bb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b48c: skipped 000723 1787848259.380984 mapread 0x002235 thru 0x004fcf (0x2d9b bytes) 000725 1787848259.410249 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b48c): Mirror 2: 0x3cd9b3de (stale) Mirror 4: 0x3cd9b3de (stale) Mirror 5: 0x3cd9b3de (stale) Mirror 6: 0xfee04e61 (stale) Mirror 7: 0x9d26f051 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b48c): Mirror 2: 0x3cd9b3de (stale) Mirror 4: 0x3cd9b3de (stale) Mirror 5: 0x3cd9b3de (stale) Mirror 6: 0xfee04e61 (stale) Mirror 7: 0x9d26f051 (stale) CRC-32 checksum value for chunk [0, 0x1b48c): Mirror 1: 0x47cb7bb6 Mirror 8: 0x47cb7bb6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b48c: skipped 000726 1787848260.329252 MIRROR_RESYNC 7 mirrors 000729 1787848262.506144 mapwrite 0x03c241 thru 0x03ffff (0x3dbf bytes) 000730 1787848262.808949 write 0x00a2be thru 0x018778 (0xe4bb bytes) 000732 1787848262.838333 mapwrite 0x010dbb thru 0x01dc21 (0xce67 bytes) 000734 1787848262.932132 trunc from 0x040000 to 0x02d3e5 (0x12c1a bytes) 000737 1787848262.997292 mapread 0x0225d2 thru 0x022e69 (0x898 bytes) 000738 1787848263.021938 write 0x026da9 thru 0x034ed0 (0xe128 bytes) 000739 1787848263.023357 mapread 0x000f61 thru 0x007e1d (0x6ebd bytes) 000740 1787848263.055218 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x34ed1): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ed1): Mirror 2: 0xa4548bc4 (stale) Mirror 4: 0xa4548bc4 (stale) Mirror 5: 0xa4548bc4 (stale) Mirror 6: 0xa4548bc4 (stale) Mirror 7: 0xa4548bc4 (stale) Mirror 8: 0xa4548bc4 (stale) CRC-32 checksum value for chunk [0, 0x34ed1): Mirror 1: 0x69ade0b4 Mirror 9: 0x69ade0b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ed1: skipped 000741 1787848264.399698 read 0x020c1d thru 0x0279b0 (0x6d94 bytes) 000742 1787848264.418932 mapwrite 0x024c89 thru 0x0263cc (0x1744 bytes) 000743 1787848264.694006 read 0x02691c thru 0x028e28 (0x250d bytes) 000744 1787848264.694611 mapread 0x00d84d thru 0x00fbca (0x237e bytes) 000745 1787848264.722554 write 0x0226cc thru 0x0245d5 (0x1f0a bytes) 000748 1787848264.728616 mapread 0x00eec9 thru 0x012bab (0x3ce3 bytes) 000750 1787848264.730181 mapread 0x007ae4 thru 0x0144d7 (0xc9f4 bytes) 000751 1787848264.736338 trunc from 0x034ed1 to 0x01c8fb (0x185d5 bytes) 000753 1787848264.818120 read 0x002562 thru 0x00cebb (0xa95a bytes) 000755 1787848264.822727 trunc from 0x01c8fc to 0x01e7c3 (0x1ec8 bytes) 000757 1787848264.865431 mapread 0x00f026 thru 0x0192fe (0xa2d9 bytes) 000758 1787848264.867315 mapread 0x0031b6 thru 0x00c8f6 (0x9741 bytes) 000759 1787848264.868776 read 0x00e485 thru 0x01d720 (0xf29c bytes) 000760 1787848264.880495 mapwrite 0x03bd3a thru 0x03ffff (0x42c6 bytes) 000762 1787848265.049494 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b9f6d75 (stale) Mirror 4: 0x4b9f6d75 (stale) Mirror 6: 0x4b9f6d75 (stale) Mirror 7: 0x4b9f6d75 (stale) Mirror 8: 0x4b9f6d75 (stale) Mirror 9: 0x5dc9ac8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b9f6d75 (stale) Mirror 4: 0x4b9f6d75 (stale) Mirror 6: 0x4b9f6d75 (stale) Mirror 7: 0x4b9f6d75 (stale) Mirror 8: 0x4b9f6d75 (stale) Mirror 9: 0x5dc9ac8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000763 1787848265.783817 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b9f6d75 (stale) Mirror 4: 0x4b9f6d75 (stale) Mirror 6: 0x4b9f6d75 (stale) Mirror 7: 0x4b9f6d75 (stale) Mirror 8: 0x4b9f6d75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b9f6d75 (stale) Mirror 4: 0x4b9f6d75 (stale) Mirror 6: 0x4b9f6d75 (stale) Mirror 7: 0x4b9f6d75 (stale) Mirror 8: 0x4b9f6d75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000765 1787848266.424864 mapwrite 0x03bcbe thru 0x03ffff (0x4342 bytes) 000768 1787848266.531145 write 0x01c808 thru 0x024db1 (0x85aa bytes) 000770 1787848266.546091 mapread 0x020fb5 thru 0x02d558 (0xc5a4 bytes) 000771 1787848266.564511 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4b9f6d75 (stale) Mirror 6: 0x4b9f6d75 (stale) Mirror 7: 0x4b9f6d75 (stale) Mirror 8: 0x4b9f6d75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4b9f6d75 (stale) Mirror 6: 0x4b9f6d75 (stale) Mirror 7: 0x4b9f6d75 (stale) Mirror 8: 0x4b9f6d75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000774 1787848267.258266 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4b9f6d75 (stale) Mirror 6: 0x4b9f6d75 (stale) Mirror 7: 0x4b9f6d75 (stale) Mirror 8: 0x4b9f6d75 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9cf46e0c Mirror 9: 0x9cf46e0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000775 1787848268.981305 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4b9f6d75 (stale) Mirror 6: 0x4b9f6d75 (stale) Mirror 7: 0x4b9f6d75 (stale) Mirror 8: 0x4b9f6d75 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9cf46e0c Mirror 9: 0x9cf46e0c Mirror 10: 0x9cf46e0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000777 1787848270.736330 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4b9f6d75 (stale) Mirror 6: 0x4b9f6d75 (stale) Mirror 7: 0x4b9f6d75 (stale) Mirror 8: 0x4b9f6d75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x4b9f6d75 (stale) Mirror 6: 0x4b9f6d75 (stale) Mirror 7: 0x4b9f6d75 (stale) Mirror 8: 0x4b9f6d75 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9cf46e0c Mirror 9: 0x9cf46e0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000781 1787848271.603303 mapread 0x026101 thru 0x02bc63 (0x5b63 bytes) 000783 1787848271.631703 mapwrite 0x00dda8 thru 0x01b1d6 (0xd42f bytes) 000785 1787848271.888925 trunc from 0x040000 to 0x03b746 (0x48b9 bytes) 000786 1787848271.928566 trunc from 0x03b747 to 0x01e3d7 (0x1d36f bytes) 000787 1787848271.957142 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e3d8): Mirror 4: 0x356d8f9a (stale) Mirror 6: 0x356d8f9a (stale) Mirror 8: 0x356d8f9a (stale) Mirror 9: 0x30dd3164 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e3d8): Mirror 4: 0x356d8f9a (stale) Mirror 6: 0x356d8f9a (stale) Mirror 8: 0x356d8f9a (stale) Mirror 9: 0x30dd3164 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e3d8: skipped 000788 1787848272.494915 read 0x002cf9 thru 0x008597 (0x589f bytes) 000789 1787848272.508857 mapread 0x009ea7 thru 0x01627e (0xc3d8 bytes) 000790 1787848272.545864 read 0x014461 thru 0x01e3d7 (0x9f77 bytes) 000792 1787848272.558961 mapread 0x00d300 thru 0x01378a (0x648b bytes) 000794 1787848272.561353 mapread 0x0137f5 thru 0x01e3d7 (0xabe3 bytes) 000796 1787848272.562995 mapwrite 0x031f62 thru 0x035955 (0x39f4 bytes) 000797 1787848272.697708 mapwrite 0x00d94b thru 0x00e7c1 (0xe77 bytes) 000800 1787848272.763206 read 0x029c60 thru 0x0339df (0x9d80 bytes) 000801 1787848272.770239 mapread 0x024802 thru 0x033df8 (0xf5f7 bytes) 000802 1787848272.790352 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35956): Mirror 6: 0xcd79550b (stale) Mirror 8: 0xcd79550b (stale) Mirror 9: 0xe4dc09bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35956): Mirror 6: 0xcd79550b (stale) Mirror 8: 0xcd79550b (stale) Mirror 9: 0xe4dc09bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35956: skipped 000803 1787848273.248499 write 0x017673 thru 0x0235bb (0xbf49 bytes) 000808 1787848273.261874 read 0x001be5 thru 0x005889 (0x3ca5 bytes) 000809 1787848273.271783 trunc from 0x035956 to 0x03bc0b (0x62b6 bytes) 000810 1787848273.295943 trunc from 0x03bc0c to 0x031b62 (0xa0a9 bytes) 000811 1787848273.320762 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x31b63): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b63): Mirror 6: 0xde98296c (stale) Mirror 8: 0xde98296c (stale) Mirror 9: 0x60bb44cf (stale) CRC-32 checksum value for chunk [0, 0x31b63): Mirror 1: 0xbd403000 Mirror 10: 0xbd403000 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b63: skipped 000813 1787848274.819134 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31b63): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31b63): Mirror 6: 0xde98296c (stale) Mirror 8: 0xde98296c (stale) Mirror 9: 0x60bb44cf (stale) CRC-32 checksum value for chunk [0, 0x31b63): Mirror 1: 0xbd403000 Mirror 10: 0xbd403000 Mirror 11: 0xbd403000 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31b63: skipped 000814 1787848275.919174 mapwrite 0x02dc86 thru 0x02f998 (0x1d13 bytes) 000815 1787848276.210053 trunc from 0x031b63 to 0x0360e9 (0x4587 bytes) 000816 1787848276.247979 mapwrite 0x00655d thru 0x00c2b8 (0x5d5c bytes) 000817 1787848276.476126 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x360ea): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x360ea): Mirror 6: 0xaf5cd0ee (stale) Mirror 8: 0xaf5cd0ee (stale) Mirror 9: 0xd753c9bb (stale) Mirror 10: 0xe0db69a8 (stale) Mirror 11: 0xe0db69a8 (stale) CRC-32 checksum value for chunk [0, 0x360ea): Mirror 1: 0x710eb498 Mirror 12: 0x710eb498 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x360ea: skipped 000819 1787848278.047147 read 0x0074a4 thru 0x00c471 (0x4fce bytes) 000820 1787848278.056411 trunc from 0x0360ea to 0x019a2f (0x1c6ba bytes) 000821 1787848278.194509 trunc from 0x019a30 to 0x012830 (0x71ff bytes) 000823 1787848278.238573 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12831): Mirror 6: 0x686b6aeb (stale) Mirror 8: 0x686b6aeb (stale) Mirror 9: 0x15ffca5a (stale) Mirror 10: 0x4460b18 (stale) Mirror 12: 0x153a236b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12831): Mirror 6: 0x686b6aeb (stale) Mirror 8: 0x686b6aeb (stale) Mirror 9: 0x15ffca5a (stale) Mirror 10: 0x4460b18 (stale) Mirror 12: 0x153a236b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12831: skipped 000824 1787848278.758087 mapread 0x0011d3 thru 0x00bac7 (0xa8f5 bytes) 000826 1787848278.781723 mapread 0x006d15 thru 0x011e38 (0xb124 bytes) 000829 1787848278.790208 trunc from 0x012831 to 0x005fca (0xc866 bytes) 000835 1787848278.813071 read 0x00138d thru 0x003aee (0x2762 bytes) 000836 1787848278.814160 write 0x03012e thru 0x03839e (0x8271 bytes) 000838 1787848278.823487 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3839f): Mirror 6: 0x77dfcf72 (stale) Mirror 8: 0x77dfcf72 (stale) Mirror 9: 0xa195eb01 (stale) Mirror 12: 0xe977fb18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3839f): Mirror 6: 0x77dfcf72 (stale) Mirror 8: 0x77dfcf72 (stale) Mirror 9: 0xa195eb01 (stale) Mirror 12: 0xe977fb18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3839f: skipped 000839 1787848279.241095 write 0x01f663 thru 0x02617a (0x6b18 bytes) 000841 1787848279.254040 write 0x0101f7 thru 0x016b6a (0x6974 bytes) 000842 1787848279.265726 write 0x020cfe thru 0x026756 (0x5a59 bytes) 000843 1787848279.267154 read 0x0207b4 thru 0x028dbc (0x8609 bytes) 000844 1787848279.273191 read 0x02f5e0 thru 0x02fab9 (0x4da bytes) 000845 1787848279.284906 trunc from 0x03839f to 0x02252c (0x15e72 bytes) 000846 1787848279.306239 mapwrite 0x02582a thru 0x03179f (0xbf76 bytes) 000847 1787848279.474695 trunc from 0x0317a0 to 0x00ca36 (0x24d69 bytes) 000850 1787848279.495695 read 0x00267a thru 0x009f34 (0x78bb bytes) 000853 1787848279.497599 read 0x004bc3 thru 0x00ca36 (0x7e74 bytes) 000857 1787848279.501762 mapwrite 0x01d23d thru 0x02b3c9 (0xe18d bytes) 000858 1787848279.605326 mapread 0x010d94 thru 0x01f1e4 (0xe451 bytes) 000859 1787848279.608125 mapwrite 0x01986b thru 0x01e1f9 (0x498f bytes) 000860 1787848279.660498 mapread 0x015563 thru 0x018876 (0x3314 bytes) 000862 1787848279.666026 trunc from 0x02b3ca to 0x014897 (0x16b32 bytes) 000863 1787848279.687543 read 0x001b2d thru 0x005aad (0x3f81 bytes) 000867 1787848279.695971 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14898): Mirror 6: 0x799a8693 (stale) Mirror 8: 0x799a8693 (stale) Mirror 9: 0x4d0a4f56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14898): Mirror 6: 0x799a8693 (stale) Mirror 8: 0x799a8693 (stale) Mirror 9: 0x4d0a4f56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14898: skipped 000868 1787848280.094662 mapread 0x00b544 thru 0x01122e (0x5ceb bytes) 000869 1787848280.114728 read 0x001376 thru 0x00adec (0x9a77 bytes) 000871 1787848280.125492 mapwrite 0x00d9ac thru 0x01be7f (0xe4d4 bytes) 000873 1787848280.305462 mapread 0x002934 thru 0x01120a (0xe8d7 bytes) 000874 1787848280.311279 mapread 0x015be3 thru 0x01be7f (0x629d bytes) 000878 1787848280.325080 trunc from 0x01be80 to 0x011e7e (0xa001 bytes) 000879 1787848280.378903 read 0x00218c thru 0x00550d (0x3382 bytes) 000880 1787848280.380090 trunc from 0x011e7f to 0x009dae (0x80d0 bytes) 000881 1787848280.436604 trunc from 0x009daf to 0x033ea2 (0x2a0f4 bytes) 000883 1787848280.489099 mapwrite 0x01a107 thru 0x021d4a (0x7c44 bytes) 000885 1787848280.644504 trunc from 0x033ea3 to 0x0075ef (0x2c8b3 bytes) 000887 1787848280.700571 mapread 0x00601f thru 0x0075ef (0x15d1 bytes) 000891 1787848280.709710 read 0x0071c5 thru 0x0075ef (0x42b bytes) 000892 1787848280.713343 trunc from 0x0075f0 to 0x00ea77 (0x7488 bytes) 000893 1787848280.741594 trunc from 0x00ea78 to 0x003060 (0xba17 bytes) 000897 1787848280.788174 write 0x02c814 thru 0x035721 (0x8f0e bytes) 000898 1787848280.791262 write 0x02a84a thru 0x032e1c (0x85d3 bytes) 000899 1787848280.800777 read 0x00ae48 thru 0x013553 (0x870c bytes) 000900 1787848280.802738 mapread 0x02f26d thru 0x035721 (0x64b5 bytes) 000902 1787848280.804975 read 0x0111cf thru 0x01c7c2 (0xb5f4 bytes) 000903 1787848280.805794 write 0x03209e thru 0x03ad19 (0x8c7c bytes) 000904 1787848280.808756 read 0x021296 thru 0x02ee3d (0xdba8 bytes) 000905 1787848280.809697 trunc from 0x03ad1a to 0x03fcc0 (0x4fa7 bytes) 000906 1787848280.836686 trunc from 0x03fcc1 to 0x02b88c (0x14434 bytes) 000907 1787848280.864758 read 0x000e16 thru 0x004da8 (0x3f93 bytes) 000910 1787848280.882254 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b88d): Mirror 8: 0x95e871d (stale) Mirror 9: 0xc3215685 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b88d): Mirror 8: 0x95e871d (stale) Mirror 9: 0xc3215685 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b88d: skipped 000913 1787848281.384451 mapread 0x015671 thru 0x0202a3 (0xac33 bytes) 000914 1787848281.435258 write 0x027f23 thru 0x02ae78 (0x2f56 bytes) 000917 1787848281.442925 mapwrite 0x022535 thru 0x0228a2 (0x36e bytes) 000918 1787848281.541557 trunc from 0x02b88d to 0x010889 (0x1b003 bytes) 000919 1787848281.564160 mapwrite 0x00f27b thru 0x010f19 (0x1c9f bytes) 000920 1787848281.724430 write 0x00bb94 thru 0x010a4a (0x4eb7 bytes) 000923 1787848281.733010 trunc from 0x010f1a to 0x01230e (0x13f5 bytes) 000930 1787848281.768643 write 0x001632 thru 0x0111de (0xfbad bytes) 000931 1787848281.788224 mapread 0x00245a thru 0x007808 (0x53af bytes) 000934 1787848281.792445 mapread 0x01199e thru 0x01230e (0x971 bytes) 000937 1787848281.808993 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1230f): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1230f): Mirror 8: 0xe8e6fe52 (stale) Mirror 9: 0x4ef41055 (stale) CRC-32 checksum value for chunk [0, 0x1230f): Mirror 1: 0x828571e2 Mirror 10: 0x828571e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1230f: skipped 000938 1787848283.060291 mapread 0x000c64 thru 0x00c5a4 (0xb941 bytes) 000939 1787848283.106550 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1230f): Mirror 8: 0xe8e6fe52 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1230f): Mirror 8: 0xe8e6fe52 (stale) CRC-32 checksum value for chunk [0, 0x1230f): Mirror 1: 0x828571e2 Mirror 10: 0x828571e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1230f: skipped 000940 1787848283.814469 MIRROR_RESYNC 3 mirrors 000943 1787848284.971944 write 0x02975c thru 0x0351ba (0xba5f bytes) 000944 1787848285.115047 mapread 0x00d1b7 thru 0x0181f1 (0xb03b bytes) 000945 1787848285.209265 read 0x032e7d thru 0x0351ba (0x233e bytes) 000947 1787848285.210301 write 0x00a51e thru 0x017dcb (0xd8ae bytes) 000949 1787848285.213252 read 0x0149bb thru 0x022cf9 (0xe33f bytes) 000951 1787848285.229171 write 0x012fc2 thru 0x01b710 (0x874f bytes) 000952 1787848285.239631 mapwrite 0x00206e thru 0x00e0a5 (0xc038 bytes) 000953 1787848285.302177 trunc from 0x0351bb to 0x02cd91 (0x8429 bytes) 000956 1787848285.344262 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cd92): Mirror 10: 0x811d44eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd92): Mirror 10: 0x811d44eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd92: skipped 000958 1787848285.939205 trunc from 0x02cd92 to 0x003d22 (0x2906f bytes) 000959 1787848285.968368 read 0x001b2c thru 0x003d22 (0x21f7 bytes) 000960 1787848285.977058 write 0x0263d0 thru 0x028f53 (0x2b84 bytes) 000962 1787848285.985188 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x176:0x0]: Invalid argument (22) 000963 1787848286.183727 write 0x00be7f thru 0x016277 (0xa3f9 bytes) 000964 1787848286.196303 mapwrite 0x029bf3 thru 0x02fd1b (0x6129 bytes) 000965 1787848286.299372 mapwrite 0x0260cd thru 0x030336 (0xa26a bytes) 000967 1787848286.384701 mapread 0x0105bf thru 0x010c1b (0x65d bytes) 000969 1787848286.386992 read 0x02992c thru 0x030336 (0x6a0b bytes) 000970 1787848286.387750 mapread 0x00a16d thru 0x00db7f (0x3a13 bytes) 000974 1787848286.401069 mapwrite 0x00feea thru 0x01fafd (0xfc14 bytes) 000976 1787848286.473020 mapwrite 0x00b11d thru 0x00c6be (0x15a2 bytes) 000979 1787848286.545998 trunc from 0x030337 to 0x0316b7 (0x1381 bytes) 000983 1787848286.587938 mapwrite 0x004add thru 0x0127f2 (0xdd16 bytes) 000985 1787848286.662363 mapwrite 0x01540e thru 0x01b220 (0x5e13 bytes) 000987 1787848286.731793 MIRROR_EXTEND to 2 mirrors 000990 1787848287.578861 mapread 0x000f01 thru 0x005eb4 (0x4fb4 bytes) 000991 1787848287.621125 read 0x030453 thru 0x0316b7 (0x1265 bytes) 000992 1787848287.634849 mapread 0x010f25 thru 0x01f036 (0xe112 bytes) 000994 1787848287.658299 mapread 0x02dcb8 thru 0x0316b7 (0x3a00 bytes) 000995 1787848287.675286 read 0x02b55b thru 0x02c232 (0xcd8 bytes) 000996 1787848287.676161 trunc from 0x0316b8 to 0x03130d (0x3aa bytes) 000997 1787848287.788911 mapread 0x00c025 thru 0x01212d (0x6109 bytes) 000998 1787848287.816303 write 0x021b71 thru 0x030c82 (0xf112 bytes) 000999 1787848287.825523 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3130e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3130e): Mirror 2: 0xee887248 (stale) CRC-32 checksum value for chunk [0, 0x3130e): Mirror 1: 0x8355a1c2 Mirror 3: 0x8355a1c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3130e: skipped 001000 1787848288.861317 read 0x0063af thru 0x012bca (0xc81c bytes) All operations completed A-OK! PASS 70a (103s) == sanity-flr test 71: check mirror extend parent fid ==== 12:31:35 (1787848295) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 12:31:39 (1787848299) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.361818 s, 5.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252223 s, 4.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 12:31:50 (1787848310) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre2 Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8113280 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2058048 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3790176 .. failed Extending file size to 8665568 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7499552 .. Extending file size to 8586752 .. Extending file size to 8771200 .. Extending file size to 8591072 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9333184 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9176960 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58591 58592 58594 58595 58599 done 192.168.204.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg402-client.virtnet /mnt/lustre2 (opts:) 192.168.204.102@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg402-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:33:12 (1787848392) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre2 Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6745952 .. Extending file size to 7978784 .. done Extending file size to 9012928 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8692128 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9015232 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 60068 60069 60071 60073 60075 failed 192.168.204.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg402-client.virtnet /mnt/lustre2 (opts:) 192.168.204.102@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg402-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:34:30 (1787848470) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137672 s, 15.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.342876 s, 6.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x98cf4771 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg402-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:34:50 (1787848490) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.112885 s, 18.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 12:34:58 (1787848498) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bb:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:35:05 (1787848505) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.209783 s, 10.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:35:15 (1787848515) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.313923 s, 6.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b4:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:35:24 (1787848524) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24043 s, 8.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:35:58 (1787848558) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg402-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134967 s, 7.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b7:0x0] } PASS 204d (63s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:37:01 (1787848621) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg402-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.391431 s, 5.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (64s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:38:05 (1787848685) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg402-server mds-ost sync done. Creating new pool test_204f oleg402-server: Pool lustre.test_204f created Adding targets to pool oleg402-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg402-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.373683 s, 5.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg402-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg402-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg402-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:39:28 (1787848768) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:39:38 (1787848778) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:39:47 (1787848787) oleg402-server: Pool lustre.test_206 created Adding targets to pool oleg402-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg402-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg402-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg402-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg402-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:40:15 (1787848815) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114819 s, 9.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966634 s, 10.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9e338c4d Mirror 3: 0x9e338c4d PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:40:29 (1787848829) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:40:33 (1787848833) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:40:37 (1787848837) Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.164235 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x17d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1180000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1180000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1180000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2470000 Read this (4 bytes): BBBB 192.168.204.102@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg402-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:41:15 (1787848875) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.021972 s, 373 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:41:23 (1787848883) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604228 s, 17.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e0 lmm_fid: [0x200000404:0x1e0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1e0 lmm_fid: [0x200000404:0x1e0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834400 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 12:41:24.000000000 -0400 Modify: 2026-08-27 12:41:25.000000000 -0400 Change: 2026-08-27 12:41:26.000000000 -0400 Birth: 2026-08-27 12:41:24.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:41:32 (1787848892) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:41:35 (1787848895) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.608068 s, 67.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0621455 s, 65.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 12:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e2:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 12:41 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:41:45 (1787848905) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:41:54 (1787848914) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 1940 sec =========== 12:42:05 (1787848925) === sanity-flr: start cleanup 12:42:07 (1787848927) === === sanity-flr: finish cleanup 12:42:14 (1787848934) ===