-----============= acceptance-small: sanity-flr ============----- Mon Sep 7 22:38:06 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg453-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:38:24 (1788835104) === oleg453-client.virtnet: executing check_config_client /mnt/lustre oleg453-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg453-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88a3d8890000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88a3d8890000.idle_timeout=debug disable quota as required oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:38:58 (1788835138) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7616] == sanity-flr test 0a: lfs mirror create with -N option == 22:39:00 (1788835140) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:39:18 (1788835158) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:39:22 (1788835162) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:39:26 (1788835166) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:39:51 (1788835191) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:39:55 (1788835195) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 22:39:59 (1788835199) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88a3d8890000.stats=clear osc.lustre-OST0001-osc-ffff88a3d8890000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:40:06 (1788835206) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:40:15 (1788835215) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:40:23 (1788835223) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:40:27 (1788835227) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:40:35 (1788835235) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 22:40:43 (1788835243) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:40:50 (1788835250) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:40:53 (1788835253) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:41:00 (1788835260) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:41:10 (1788835270) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:41:17 (1788835277) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 1.17397 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211934 s, 4.9 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:41:29 (1788835289) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.128974 s, 56.9 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:41:40 (1788835300) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:41:51 (1788835311) Stopping /mnt/lustre-ost1 (opts:) on oleg453-server oleg453-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88a3d8890000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88a3d8890000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg453-server: oleg453-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg453-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg453-server oleg453-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88a3d8890000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88a3d8890000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg453-server: oleg453-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg453-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (70s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:43:01 (1788835381) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.58538 s, 14.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg453-server oleg453-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88a3d8890000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88a3d8890000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg453-server: oleg453-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg453-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:43:46 (1788835426) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg453-server oleg453-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88a3d8890000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88a3d8890000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg453-server: oleg453-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg453-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg453-server oleg453-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88a3d8890000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88a3d8890000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg453-server: oleg453-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg453-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:45:04 (1788835504) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.68566 s, 11.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 7 22:45 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.899s user 0m0.001s sys 0m0.178s Stopping /mnt/lustre-ost1 (opts:) on oleg453-server oleg453-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88a3d8890000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88a3d8890000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.087s user 0m0.000s sys 0m0.203s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg453-server: oleg453-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg453-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg453-server oleg453-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88a3d8890000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88a3d8890000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.310s user 0m0.001s sys 0m0.190s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg453-server: oleg453-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg453-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg453-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88a3d8890000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88a3d8890000.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:46:37 (1788835597) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:46:39 (1788835599) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:46:42 (1788835602) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 22:46:45 (1788835605) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 22:46:53 (1788835613) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.568979 s, 22.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:47:07 (1788835627) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368457 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542651 s, 19.3 MB/s PASS 36b (11s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:47:18 (1788835638) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:47:27 (1788835647) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000811972 s, 1.3 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000807355 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000827069 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00322313 s, 318 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 22:47:39 (1788835659) create test files with size 15742424 .. 2509c7f3a0c83335520cb34fd9e54f50 - 393e201f5c6e4ff5b5066f710bd1b550 - 2d1cd95e00c04e19ddee60cf2dce04d0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg453-client.virtnet /mnt/lustre (opts:) Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 22:48:02 (1788835682) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:48:16 (1788835696) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:48:23 (1788835703) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102213 s, 10.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368287 s, 28.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 22:48:33 (1788835713) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193662 s, 10.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.484063 s, 8.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.313735 s, 6.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 22:48:50 (1788835730) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 22:48:54 (1788835734) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:48:57 (1788835737) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320842 s, 32.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 22:49:05 (1788835745) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:49:09 (1788835749) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:49:19 (1788835759) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.361912 s, 29.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg453-client.virtnet /mnt/lustre (opts:) Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:49:31 (1788835771) llite.lustre-ffff88a3c3100800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.229902 s, 45.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg453-client.virtnet /mnt/lustre (opts:) Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff88a3e1062000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:49:44 (1788835784) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:50:29 (1788835829) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:51:10 (1788835870) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:51:13 (1788835873) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.267099 s, 11.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.124568 s, 25.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xe1fa7553 (nosync,stale) Mirror 2: 0xe1fa7553 (nosync,stale) ** verify mirror 2 doesn't change original checksum: fdd09b321658a6a8fb2f080c27f26572 - mirror 1 checksum: 3519ae3f07fb7f195377d9e97eb5798e - mirror 2 checksum: fdd09b321658a6a8fb2f080c27f26572 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T22:51:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-07T22:51:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: fdd09b321658a6a8fb2f080c27f26572 - mirror 1 checksum: 3519ae3f07fb7f195377d9e97eb5798e - mirror 2 checksum: 3519ae3f07fb7f195377d9e97eb5798e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T22:51:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-07T22:51:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:51:27 (1788835887) Failing mds1 on oleg453-server Stopping /mnt/lustre-mds1 (opts:) on oleg453-server 22:51:35 (1788835895) shut down facet: mds1 facet_host: oleg453-server facet_failover_host: oleg453-server Failover mds1 to oleg453-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Started lustre-MDT0000 22:51:58 (1788835918) targets are mounted 22:51:58 (1788835918) facet_failover done oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262343 s, 40.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:52:14 (1788835934) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:52:17 (1788835937) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:52:20 (1788835940) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0120085 s, 1.4 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:52:28 (1788835948) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:52:31 (1788835951) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00601607 s, 681 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0528397 s, 77.5 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:52:38 (1788835958) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:52:58 (1788835978) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:53:26 (1788836006) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:53:49 (1788836029) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:54:12 (1788836052) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 22:54:19 (1788836059) mirror create pid 54320 mirror split pid 54321 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54320 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54321 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 22:55:28 (1788836128) Seed set to 6131 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1788836131.796011 mapwrite 0x01dd75 thru 0x021b7c (0x3e08 bytes) 000002 1788836131.960539 read 0x01ef6d thru 0x021b7c (0x2c10 bytes) 000003 1788836131.961616 punch from 0x017cbf to 0x021b7c (0x9ebe bytes) 000004 1788836131.969776 mapwrite 0x010ba0 thru 0x0200b1 (0xf512 bytes) truncating to largest ever: 0x2ad7e 000005 1788836132.055455 trunc from 0x021b7d to 0x02ad7d (0x9201 bytes) 000006 1788836132.071916 trunc from 0x02ad7e to 0x00b0b3 (0x1fcca bytes) 000007 1788836132.091258 mapwrite 0x03347f thru 0x03c89e (0x9420 bytes) 000010 1788836132.198387 punch from 0x01c067 to 0x01d3b9 (0x1353 bytes) 000011 1788836132.208600 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x2d756 000012 1788836133.500917 trunc from 0x03c89f to 0x02d755 (0xf149 bytes) 000013 1788836133.616687 trunc from 0x02d756 to 0x01b07a (0x126db bytes) 000014 1788836133.653208 punch from 0x003245 to 0x00d3f6 (0xa1b2 bytes) 000015 1788836133.670585 trunc from 0x01b07b to 0x01d074 (0x1ffa bytes) 000017 1788836133.717793 trunc from 0x01d075 to 0x02c4d2 (0xf45e bytes) 000018 1788836133.762441 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c4d3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c4d3): Mirror 2: 0xd4df9afa (stale) CRC-32 checksum value for chunk [0, 0x2c4d3): Mirror 1: 0xd4df9afa Mirror 3: 0xd4df9afa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c4d3: skipped 000019 1788836135.609804 read 0x0059f4 thru 0x00bdb1 (0x63be bytes) 000021 1788836135.631816 mapwrite 0x014cc4 thru 0x01d8d2 (0x8c0f bytes) 000023 1788836135.865307 write 0x008a3b thru 0x015af3 (0xd0b9 bytes) 000024 1788836135.875482 punch from 0x0245b4 to 0x02c4d2 (0x7f1f bytes) 000025 1788836135.894579 mapwrite 0x01abe1 thru 0x01e94f (0x3d6f bytes) 000027 1788836135.967289 mapread 0x011df4 thru 0x019373 (0x7580 bytes) 000029 1788836135.970552 mapread 0x0286ba thru 0x02c4d2 (0x3e19 bytes) 000030 1788836136.008663 mapread 0x021920 thru 0x021e35 (0x516 bytes) 000033 1788836136.012683 write 0x006689 thru 0x012e0c (0xc784 bytes) 000034 1788836136.028264 mapwrite 0x02481b thru 0x032e82 (0xe668 bytes) 000036 1788836136.378418 read 0x026fe5 thru 0x032205 (0xb221 bytes) 000037 1788836136.379010 mapread 0x01fe38 thru 0x024ddf (0x4fa8 bytes) 000038 1788836136.394949 write 0x023b00 thru 0x03220e (0xe70f bytes) 000039 1788836136.403459 mapwrite 0x00d646 thru 0x01139e (0x3d59 bytes) 000042 1788836136.554394 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32e83): Mirror 2: 0x44859494 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32e83): Mirror 2: 0x44859494 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32e83: skipped 000043 1788836136.916553 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000044 1788836137.103981 punch from 0x001878 to 0x007ce3 (0x646c bytes) 000045 1788836137.112689 write 0x026fb0 thru 0x02ebbc (0x7c0d bytes) 000046 1788836137.139947 read 0x00bb34 thru 0x0179e2 (0xbeaf bytes) 000048 1788836137.159510 read 0x022d7f thru 0x02eb68 (0xbdea bytes) 000049 1788836137.174322 punch from 0x014717 to 0x014e00 (0x6ea bytes) 000050 1788836137.194656 read 0x001f6c thru 0x00a498 (0x852d bytes) 000051 1788836137.195513 mapread 0x02fe12 thru 0x031f6b (0x215a bytes) 000052 1788836137.197457 mapwrite 0x01fc30 thru 0x02f514 (0xf8e5 bytes) 000055 1788836137.325569 write 0x034731 thru 0x03ffff (0xb8cf bytes) 000056 1788836137.330296 write 0x0374b3 thru 0x03a71d (0x326b bytes) 000057 1788836137.331584 punch from 0x01b401 to 0x023c3a (0x883a bytes) 000058 1788836137.375934 punch from 0x034583 to 0x03ffff (0xba7d bytes) 000059 1788836137.384771 punch from 0x022fd5 to 0x02925e (0x628a bytes) 000060 1788836137.395448 mapread 0x03162d thru 0x032475 (0xe49 bytes) 000061 1788836137.399747 write 0x033b83 thru 0x03ffff (0xc47d bytes) 000062 1788836137.416537 read 0x004609 thru 0x0046e2 (0xda bytes) 000063 1788836137.417473 mapwrite 0x001150 thru 0x00a5df (0x9490 bytes) 000064 1788836137.518135 trunc from 0x040000 to 0x017d27 (0x282d8 bytes) 000066 1788836137.552748 mapread 0x013a9a thru 0x017d27 (0x428e bytes) 000067 1788836137.553890 mapwrite 0x00e393 thru 0x01d9f0 (0xf65e bytes) 000068 1788836137.699287 punch from 0x00329e to 0x00e1bc (0xaf1f bytes) 000069 1788836137.708857 punch from 0x019507 to 0x01d9f0 (0x44ea bytes) 000070 1788836137.716462 mapwrite 0x021f34 thru 0x02560b (0x36d8 bytes) 000071 1788836137.851917 read 0x004c8a thru 0x006b55 (0x1ecc bytes) 000072 1788836137.866401 trunc from 0x02560c to 0x02d065 (0x7a5a bytes) 000073 1788836137.887386 mapwrite 0x02bbdd thru 0x02cde9 (0x120d bytes) 000074 1788836138.043263 mapread 0x00b99a thru 0x01084a (0x4eb1 bytes) 000075 1788836138.045514 read 0x006b7d thru 0x0086f2 (0x1b76 bytes) 000077 1788836138.054239 write 0x015cf5 thru 0x01987d (0x3b89 bytes) 000078 1788836138.059255 mapwrite 0x018aae thru 0x01bb5c (0x30af bytes) 000079 1788836138.184889 mapwrite 0x039ffc thru 0x03b6ae (0x16b3 bytes) 000080 1788836138.341302 mapwrite 0x03d3ef thru 0x03ffff (0x2c11 bytes) 000081 1788836138.530290 trunc from 0x040000 to 0x026ac0 (0x1953f bytes) 000083 1788836138.572607 punch from 0x00ede6 to 0x01dd2b (0xef46 bytes) 000084 1788836138.579998 trunc from 0x026ac1 to 0x022a75 (0x404b bytes) 000087 1788836138.635347 mapwrite 0x014c6b thru 0x01c050 (0x73e6 bytes) 000088 1788836138.830034 mapwrite 0x0149b8 thru 0x021dff (0xd448 bytes) 000090 1788836138.974279 mapwrite 0x01670f thru 0x01e4e5 (0x7dd7 bytes) 000093 1788836139.080094 trunc from 0x022a76 to 0x019826 (0x924f bytes) 000094 1788836139.106363 mapwrite 0x01da7a thru 0x024e0c (0x7393 bytes) truncating to largest ever: 0x38c24 000096 1788836139.302089 trunc from 0x024e0d to 0x038c23 (0x13e17 bytes) 000097 1788836139.319720 trunc from 0x038c24 to 0x024e6b (0x13db8 bytes) 000101 1788836139.349713 mapwrite 0x005526 thru 0x01343e (0xdf19 bytes) 000102 1788836139.487464 write 0x019b03 thru 0x01e354 (0x4852 bytes) 000103 1788836139.490887 write 0x017c3f thru 0x024082 (0xc444 bytes) 000105 1788836139.493335 write 0x006550 thru 0x00a044 (0x3af5 bytes) 000106 1788836139.495759 mapwrite 0x001407 thru 0x00a15b (0x8d55 bytes) 000108 1788836139.622292 mapread 0x01c494 thru 0x024e6b (0x89d8 bytes) 000109 1788836139.623207 read 0x00b27e thru 0x0117bb (0x653e bytes) 000110 1788836139.626463 punch from 0x00f5bb to 0x011555 (0x1f9b bytes) 000112 1788836139.635589 write 0x019056 thru 0x01f05f (0x600a bytes) 000113 1788836139.638990 mapread 0x01ce07 thru 0x024e6b (0x8065 bytes) 000114 1788836139.642595 trunc from 0x024e6c to 0x0232af (0x1bbc bytes) 000115 1788836139.671445 write 0x03888a thru 0x03d343 (0x4aba bytes) 000116 1788836139.672660 read 0x02c1b1 thru 0x034309 (0x8159 bytes) 000118 1788836139.684956 mapwrite 0x038f89 thru 0x03ffff (0x7077 bytes) 000119 1788836139.842119 read 0x02ac04 thru 0x02fe65 (0x5262 bytes) 000120 1788836139.842725 trunc from 0x040000 to 0x031630 (0xe9cf bytes) 000121 1788836139.866467 MIRROR_EXTEND to 2 mirrors 000124 1788836141.836111 punch from 0x013324 to 0x01d7cd (0xa4aa bytes) 000125 1788836142.083178 punch from 0x00689a to 0x00ff07 (0x966e bytes) 000126 1788836142.095872 trunc from 0x031631 to 0x0265c0 (0xb070 bytes) 000127 1788836142.145367 trunc from 0x0265c1 to 0x01581c (0x10da4 bytes) 000128 1788836142.209031 read 0x000e75 thru 0x00ec95 (0xde21 bytes) 000129 1788836142.219448 read 0x01510e thru 0x01581c (0x70f bytes) 000131 1788836142.231541 read 0x0011a8 thru 0x006be1 (0x5a3a bytes) 000134 1788836142.237378 trunc from 0x01581d to 0x0212a3 (0xba87 bytes) 000135 1788836142.257621 read 0x001858 thru 0x00502d (0x37d6 bytes) 000138 1788836142.267193 MIRROR_RESYNC 2 mirrors 000139 1788836143.676807 trunc from 0x0212a4 to 0x0373e9 (0x16146 bytes) 000140 1788836143.764659 mapread 0x02a2d0 thru 0x0360aa (0xbddb bytes) 000143 1788836143.805770 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x373ea): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x373ea): Mirror 2: 0x7c16e6dd (stale) CRC-32 checksum value for chunk [0, 0x373ea): Mirror 1: 0x7c16e6dd Mirror 3: 0x7c16e6dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x373ea: skipped 000145 1788836145.774567 trunc from 0x0373ea to 0x02904d (0xe39c bytes) 000146 1788836145.995127 read 0x010886 thru 0x01b5f1 (0xad6c bytes) 000148 1788836146.018201 trunc from 0x02904e to 0x00bc01 (0x1d44c bytes) 000149 1788836146.074669 write 0x01f59f thru 0x0292db (0x9d3d bytes) 000150 1788836146.086083 write 0x037f79 thru 0x0381f3 (0x27b bytes) 000151 1788836146.121620 read 0x02ac2f thru 0x035cb2 (0xb084 bytes) 000153 1788836146.178965 mapread 0x00990e thru 0x00e94b (0x503e bytes) 000154 1788836146.189758 punch from 0x01049b to 0x010ad0 (0x636 bytes) 000156 1788836146.247500 punch from 0x0301bc to 0x033f6c (0x3db1 bytes) 000157 1788836146.263655 punch from 0x01dfec to 0x0289aa (0xa9bf bytes) 000159 1788836146.274806 write 0x03e9da thru 0x03ffff (0x1626 bytes) 000160 1788836146.278939 mapwrite 0x0074b9 thru 0x00fc03 (0x874b bytes) 000161 1788836146.469933 read 0x01a2cf thru 0x023bda (0x990c bytes) 000162 1788836146.499088 punch from 0x004c7f to 0x00dfab (0x932d bytes) 000164 1788836146.519496 mapwrite 0x00240e thru 0x002694 (0x287 bytes) 000165 1788836146.573569 write 0x002b81 thru 0x011aa3 (0xef23 bytes) 000166 1788836146.575249 punch from 0x01fa28 to 0x025d98 (0x6371 bytes) 000167 1788836146.600245 punch from 0x02d5a4 to 0x02d868 (0x2c5 bytes) 000168 1788836146.607271 mapread 0x0295a5 thru 0x02c35e (0x2dba bytes) 000171 1788836146.610032 mapwrite 0x002d62 thru 0x0064b3 (0x3752 bytes) 000173 1788836146.676301 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1f28a880 (stale) Mirror 3: 0x1f28a880 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xcc7446fc Mirror 4: 0xcc7446fc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000174 1788836148.460732 write 0x010adb thru 0x01b945 (0xae6b bytes) 000175 1788836148.609129 write 0x02f4ce thru 0x031ad5 (0x2608 bytes) 000176 1788836148.627426 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1f28a880 (stale) Mirror 3: 0x1f28a880 (stale) Mirror 4: 0xcc7446fc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc65b0b69 Mirror 5: 0xc65b0b69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000177 1788836150.167325 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1f28a880 (stale) Mirror 3: 0x1f28a880 (stale) Mirror 4: 0xcc7446fc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc65b0b69 Mirror 5: 0xc65b0b69 Mirror 6: 0xc65b0b69 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000180 1788836151.165300 read 0x005d8f thru 0x011361 (0xb5d3 bytes) 000181 1788836151.179699 punch from 0x01d135 to 0x023834 (0x6700 bytes) 000184 1788836151.344947 mapread 0x034360 thru 0x036a17 (0x26b8 bytes) 000185 1788836151.366580 mapread 0x02c731 thru 0x039941 (0xd211 bytes) 000186 1788836151.378651 mapread 0x036a80 thru 0x03ffff (0x9580 bytes) 000187 1788836151.408930 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1f28a880 (stale) Mirror 4: 0xcc7446fc (stale) Mirror 5: 0xc65b0b69 (stale) Mirror 6: 0xc65b0b69 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1f28a880 (stale) Mirror 4: 0xcc7446fc (stale) Mirror 5: 0xc65b0b69 (stale) Mirror 6: 0xc65b0b69 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000188 1788836151.839931 mapwrite 0x01248a thru 0x012d90 (0x907 bytes) 000189 1788836151.932899 write 0x0232a5 thru 0x02a828 (0x7584 bytes) 000190 1788836151.956301 read 0x00c229 thru 0x013a40 (0x7818 bytes) 000191 1788836151.970540 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1f28a880 (stale) Mirror 4: 0xcc7446fc (stale) Mirror 5: 0xc65b0b69 (stale) Mirror 6: 0xc65b0b69 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9f36814b Mirror 7: 0x9f36814b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000192 1788836153.775634 trunc from 0x040000 to 0x02dad7 (0x12528 bytes) 000195 1788836153.931535 mapread 0x00921f thru 0x014c4b (0xba2d bytes) 000196 1788836154.014936 mapwrite 0x01eded thru 0x02837f (0x9593 bytes) 000197 1788836154.253047 write 0x02f836 thru 0x039ad2 (0xa29d bytes) 000201 1788836154.256558 trunc from 0x039ad3 to 0x0314e7 (0x85eb bytes) 000202 1788836154.290133 trunc from 0x0314e8 to 0x008627 (0x28ec0 bytes) 000203 1788836154.329122 read 0x007c33 thru 0x008627 (0x9f5 bytes) 000204 1788836154.340305 MIRROR_RESYNC 6 mirrors truncating to largest ever: 0x3adf5 000205 1788836156.248499 trunc from 0x008628 to 0x03adf4 (0x327cd bytes) 000209 1788836156.334760 mapread 0x03089e thru 0x030a08 (0x16b bytes) 000210 1788836156.343441 mapwrite 0x02ed1b thru 0x030341 (0x1627 bytes) 000211 1788836156.420540 MIRROR_RESYNC 6 mirrors 000212 1788836158.723501 write 0x01c9d4 thru 0x025083 (0x86b0 bytes) 000213 1788836158.871741 write 0x01895b thru 0x027164 (0xe80a bytes) 000214 1788836158.888479 mapread 0x034c65 thru 0x039b0a (0x4ea6 bytes) 000215 1788836158.929630 read 0x02f4a2 thru 0x03adf4 (0xb953 bytes) 000216 1788836158.930296 mapwrite 0x0134b0 thru 0x02259f (0xf0f0 bytes) 000217 1788836159.021398 punch from 0x012469 to 0x020a43 (0xe5db bytes) 000218 1788836159.028048 mapread 0x00c142 thru 0x013436 (0x72f5 bytes) 000219 1788836159.046517 trunc from 0x03adf5 to 0x02c332 (0xeac2 bytes) 000220 1788836159.075465 write 0x020abe thru 0x020bc7 (0x10a bytes) 000222 1788836159.085491 trunc from 0x02c333 to 0x01aeb9 (0x11479 bytes) 000227 1788836159.112052 punch from 0x003d22 to 0x012828 (0xeb07 bytes) 000228 1788836159.124169 mapread 0x00edde thru 0x014a7c (0x5c9f bytes) 000230 1788836159.137107 mapread 0x00aaf8 thru 0x00f3af (0x48b8 bytes) 000232 1788836159.139432 punch from 0x00f88d to 0x017ed7 (0x864b bytes) 000235 1788836159.150109 punch from 0x00141a to 0x00e56d (0xd154 bytes) 000236 1788836159.155956 mapread 0x005d93 thru 0x00e1cf (0x843d bytes) 000237 1788836159.171858 mapwrite 0x02e159 thru 0x037e06 (0x9cae bytes) 000238 1788836159.278845 punch from 0x01d5aa to 0x02b683 (0xe0da bytes) 000239 1788836159.286302 mapread 0x037027 thru 0x037e06 (0xde0 bytes) 000241 1788836159.289795 mapwrite 0x02f45b thru 0x03f01f (0xfbc5 bytes) 000242 1788836159.424910 read 0x024610 thru 0x0279d6 (0x33c7 bytes) 000244 1788836159.446987 read 0x017065 thru 0x01a702 (0x369e bytes) 000245 1788836159.447743 punch from 0x025cda to 0x0354bd (0xf7e4 bytes) 000246 1788836159.454278 mapwrite 0x000dc4 thru 0x0069fc (0x5c39 bytes) 000247 1788836159.566896 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f020): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f020): Mirror 3: 0xb80eee93 (stale) Mirror 4: 0xb80eee93 (stale) Mirror 5: 0xb80eee93 (stale) Mirror 6: 0xb80eee93 (stale) Mirror 7: 0xb80eee93 (stale) CRC-32 checksum value for chunk [0, 0x3f020): Mirror 1: 0xf411f46d Mirror 8: 0xf411f46d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f020: skipped 000248 1788836161.352386 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f020): Mirror 3: 0xb80eee93 (stale) Mirror 4: 0xb80eee93 (stale) Mirror 5: 0xb80eee93 (stale) Mirror 7: 0xb80eee93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f020): Mirror 3: 0xb80eee93 (stale) Mirror 4: 0xb80eee93 (stale) Mirror 5: 0xb80eee93 (stale) Mirror 7: 0xb80eee93 (stale) CRC-32 checksum value for chunk [0, 0x3f020): Mirror 1: 0xf411f46d Mirror 8: 0xf411f46d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f020: skipped 000249 1788836162.265869 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f020): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f020): Mirror 3: 0xb80eee93 (stale) Mirror 4: 0xb80eee93 (stale) Mirror 5: 0xb80eee93 (stale) Mirror 7: 0xb80eee93 (stale) CRC-32 checksum value for chunk [0, 0x3f020): Mirror 1: 0xf411f46d Mirror 8: 0xf411f46d Mirror 9: 0xf411f46d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f020: skipped 000251 1788836163.572604 write 0x027af5 thru 0x030bda (0x90e6 bytes) 000252 1788836163.784245 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f020): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f020): Mirror 3: 0xb80eee93 (stale) Mirror 4: 0xb80eee93 (stale) Mirror 5: 0xb80eee93 (stale) Mirror 7: 0xb80eee93 (stale) Mirror 8: 0xf411f46d (stale) Mirror 9: 0xf411f46d (stale) CRC-32 checksum value for chunk [0, 0x3f020): Mirror 1: 0x7a744d2c Mirror 10: 0x7a744d2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f020: skipped 000253 1788836165.688822 read 0x01d676 thru 0x02d301 (0xfc8c bytes) 000254 1788836165.716249 mapwrite 0x0150fd thru 0x019917 (0x481b bytes) 000255 1788836166.006715 write 0x01b0d7 thru 0x02b089 (0xffb3 bytes) 000257 1788836166.022874 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f020): Mirror 4: 0xb80eee93 (stale) Mirror 5: 0xb80eee93 (stale) Mirror 7: 0xb80eee93 (stale) Mirror 8: 0xf411f46d (stale) Mirror 9: 0xf411f46d (stale) Mirror 10: 0x7a744d2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f020): Mirror 4: 0xb80eee93 (stale) Mirror 5: 0xb80eee93 (stale) Mirror 7: 0xb80eee93 (stale) Mirror 8: 0xf411f46d (stale) Mirror 9: 0xf411f46d (stale) Mirror 10: 0x7a744d2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f020: skipped 000258 1788836166.682294 punch from 0x0329fd to 0x039b70 (0x7174 bytes) 000261 1788836166.692558 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f020): Mirror 4: 0xb80eee93 (stale) Mirror 7: 0xb80eee93 (stale) Mirror 8: 0xf411f46d (stale) Mirror 9: 0xf411f46d (stale) Mirror 10: 0x7a744d2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f020): Mirror 4: 0xb80eee93 (stale) Mirror 7: 0xb80eee93 (stale) Mirror 8: 0xf411f46d (stale) Mirror 9: 0xf411f46d (stale) Mirror 10: 0x7a744d2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f020: skipped 000262 1788836167.139843 mapread 0x031737 thru 0x03f01f (0xd8e9 bytes) 000263 1788836167.186872 write 0x027f09 thru 0x02cca6 (0x4d9e bytes) 000264 1788836167.202006 mapread 0x03a93a thru 0x03f01f (0x46e6 bytes) 000268 1788836167.214096 write 0x024882 thru 0x031001 (0xc780 bytes) 000269 1788836167.231004 punch from 0x03b8e1 to 0x03ee66 (0x3586 bytes) 000270 1788836167.276968 read 0x02536f thru 0x0262ed (0xf7f bytes) 000271 1788836167.285844 trunc from 0x03f020 to 0x02498f (0x1a690 bytes) 000273 1788836167.330721 write 0x02fa9c thru 0x02fee9 (0x44e bytes) 000275 1788836167.334478 punch from 0x01a788 to 0x022ede (0x8757 bytes) 000276 1788836167.362936 MIRROR_RESYNC 6 mirrors 000277 1788836170.473539 mapwrite 0x024407 thru 0x02b4a9 (0x70a3 bytes) 000278 1788836170.790555 MIRROR_RESYNC 6 mirrors 000280 1788836174.290969 punch from 0x00c35f to 0x015178 (0x8e1a bytes) 000281 1788836174.362670 punch from 0x000538 to 0x00f9a5 (0xf46e bytes) 000282 1788836174.368747 write 0x039241 thru 0x03ffff (0x6dbf bytes) 000283 1788836174.369973 read 0x012d17 thru 0x0192aa (0x6594 bytes) 000284 1788836174.378438 read 0x0080d4 thru 0x011b1e (0x9a4b bytes) 000286 1788836174.390810 trunc from 0x040000 to 0x00b7c2 (0x3483d bytes) 000287 1788836174.421485 punch from 0x000bd5 to 0x00b7c2 (0xabee bytes) 000288 1788836174.432931 read 0x005a96 thru 0x00b7c2 (0x5d2d bytes) 000289 1788836174.445436 punch from 0x00b613 to 0x00b7c2 (0x1b0 bytes) 000290 1788836174.459825 trunc from 0x00b7c3 to 0x03845e (0x2cc9c bytes) truncating to largest ever: 0x3f6ef 000292 1788836174.482903 trunc from 0x03845f to 0x03f6ee (0x7290 bytes) 000294 1788836174.504464 mapread 0x028c0f thru 0x02d116 (0x4508 bytes) 000295 1788836174.518761 write 0x01df1a thru 0x02dc42 (0xfd29 bytes) 000296 1788836174.525571 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3f6ef): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6ef): Mirror 4: 0xaab0c5c6 (stale) Mirror 7: 0xaab0c5c6 (stale) Mirror 8: 0xaab0c5c6 (stale) Mirror 9: 0xaab0c5c6 (stale) Mirror 10: 0xaab0c5c6 (stale) CRC-32 checksum value for chunk [0, 0x3f6ef): Mirror 1: 0x3adb670d Mirror 11: 0x3adb670d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6ef: skipped 000297 1788836176.071400 read 0x0000da thru 0x0055e6 (0x550d bytes) 000299 1788836176.088258 trunc from 0x03f6ef to 0x0057c3 (0x39f2b bytes) 000300 1788836176.259180 mapwrite 0x02e362 thru 0x02ffcf (0x1c6e bytes) 000301 1788836176.441251 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ffd0): Mirror 4: 0xa806ec87 (stale) Mirror 7: 0xa806ec87 (stale) Mirror 9: 0xa806ec87 (stale) Mirror 10: 0xa806ec87 (stale) Mirror 11: 0x2b67a1e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffd0): Mirror 4: 0xa806ec87 (stale) Mirror 7: 0xa806ec87 (stale) Mirror 9: 0xa806ec87 (stale) Mirror 10: 0xa806ec87 (stale) Mirror 11: 0x2b67a1e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffd0: skipped 000303 1788836177.114550 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ffd0): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffd0): Mirror 4: 0xa806ec87 (stale) Mirror 7: 0xa806ec87 (stale) Mirror 9: 0xa806ec87 (stale) Mirror 10: 0xa806ec87 (stale) Mirror 11: 0x2b67a1e5 (stale) CRC-32 checksum value for chunk [0, 0x2ffd0): Mirror 1: 0x1c4a32f5 Mirror 12: 0x1c4a32f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffd0: skipped 000304 1788836178.419681 mapwrite 0x0387f3 thru 0x03ffff (0x780d bytes) 000305 1788836178.786466 MIRROR_RESYNC 7 mirrors 000307 1788836181.881270 mapread 0x01a990 thru 0x01cd3e (0x23af bytes) 000308 1788836181.905369 mapwrite 0x017c9b thru 0x0242b6 (0xc61c bytes) 000309 1788836182.157791 write 0x0211ba thru 0x02d777 (0xc5be bytes) 000310 1788836182.172120 punch from 0x03ead5 to 0x03ffff (0x152b bytes) 000311 1788836182.197678 read 0x0060cd thru 0x0081bc (0x20f0 bytes) 000315 1788836182.220807 write 0x011014 thru 0x01d2dd (0xc2ca bytes) 000316 1788836182.222749 trunc from 0x040000 to 0x031585 (0xea7a bytes) 000318 1788836182.280458 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x31586): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31586): Mirror 4: 0x128415e8 (stale) Mirror 7: 0x128415e8 (stale) Mirror 9: 0x128415e8 (stale) Mirror 10: 0x128415e8 (stale) Mirror 11: 0x128415e8 (stale) Mirror 12: 0x128415e8 (stale) CRC-32 checksum value for chunk [0, 0x31586): Mirror 1: 0xfb3c7e53 Mirror 13: 0xfb3c7e53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31586: skipped 000319 1788836183.515145 mapread 0x0205fe thru 0x02f8f0 (0xf2f3 bytes) 000320 1788836183.552001 punch from 0x00bcfb to 0x019203 (0xd509 bytes) 000322 1788836183.679512 read 0x00aaf1 thru 0x018166 (0xd676 bytes) 000323 1788836183.727754 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x31586): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31586): Mirror 4: 0x128415e8 (stale) Mirror 7: 0x128415e8 (stale) Mirror 9: 0x128415e8 (stale) Mirror 10: 0x128415e8 (stale) Mirror 11: 0x128415e8 (stale) Mirror 12: 0x128415e8 (stale) Mirror 13: 0xfb3c7e53 (stale) CRC-32 checksum value for chunk [0, 0x31586): Mirror 1: 0x2f3404fd Mirror 14: 0x2f3404fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31586: skipped 000326 1788836185.921862 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31586): Mirror 4: 0x128415e8 (stale) Mirror 7: 0x128415e8 (stale) Mirror 9: 0x128415e8 (stale) Mirror 11: 0x128415e8 (stale) Mirror 12: 0x128415e8 (stale) Mirror 13: 0xfb3c7e53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31586): Mirror 4: 0x128415e8 (stale) Mirror 7: 0x128415e8 (stale) Mirror 9: 0x128415e8 (stale) Mirror 11: 0x128415e8 (stale) Mirror 12: 0x128415e8 (stale) Mirror 13: 0xfb3c7e53 (stale) CRC-32 checksum value for chunk [0, 0x31586): Mirror 1: 0x2f3404fd Mirror 14: 0x2f3404fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31586: skipped 000327 1788836187.295802 mapwrite 0x033682 thru 0x03dd42 (0xa6c1 bytes) 000330 1788836187.656887 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dd43): Mirror 4: 0x9c62684c (stale) Mirror 7: 0x9c62684c (stale) Mirror 11: 0x9c62684c (stale) Mirror 12: 0x9c62684c (stale) Mirror 13: 0x521ab54e (stale) Mirror 14: 0xeac10be2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd43): Mirror 4: 0x9c62684c (stale) Mirror 7: 0x9c62684c (stale) Mirror 11: 0x9c62684c (stale) Mirror 12: 0x9c62684c (stale) Mirror 13: 0x521ab54e (stale) Mirror 14: 0xeac10be2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd43: skipped 000332 1788836188.518981 punch from 0x03d76a to 0x03dd42 (0x5d9 bytes) 000335 1788836188.530578 mapwrite 0x01cc77 thru 0x025feb (0x9375 bytes) 000337 1788836188.642448 write 0x0110fa thru 0x020c7b (0xfb82 bytes) 000338 1788836188.653417 punch from 0x028503 to 0x0324e7 (0x9fe5 bytes) 000339 1788836188.678695 punch from 0x027bab to 0x034a06 (0xce5c bytes) 000340 1788836188.688540 write 0x017aca thru 0x019574 (0x1aab bytes) 000343 1788836188.697001 mapread 0x021064 thru 0x0223b1 (0x134e bytes) 000344 1788836188.699536 write 0x021882 thru 0x028f92 (0x7711 bytes) 000345 1788836188.711646 trunc from 0x03dd43 to 0x023f16 (0x19e2c bytes) 000346 1788836188.737523 write 0x00f50f thru 0x018d52 (0x9844 bytes) 000347 1788836188.747304 trunc from 0x023f17 to 0x0227d7 (0x173f bytes) 000349 1788836188.779879 read 0x006c7c thru 0x0141a2 (0xd527 bytes) 000350 1788836188.791611 mapread 0x00ddaf thru 0x01b979 (0xdbcb bytes) 000351 1788836188.802834 MIRROR_RESYNC 7 mirrors 000352 1788836191.919537 read 0x014222 thru 0x019df9 (0x5bd8 bytes) 000353 1788836191.930276 trunc from 0x0227d8 to 0x02a832 (0x805b bytes) 000354 1788836192.063625 mapwrite 0x0109c8 thru 0x012cf2 (0x232b bytes) 000356 1788836192.262781 MIRROR_RESYNC 7 mirrors 000357 1788836195.374598 trunc from 0x02a833 to 0x01a486 (0x103ac bytes) 000358 1788836195.574047 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1a487): Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a487): Mirror 4: 0x2767a81 (stale) Mirror 7: 0x2767a81 (stale) Mirror 11: 0x2767a81 (stale) Mirror 12: 0x2767a81 (stale) Mirror 13: 0x2767a81 (stale) Mirror 14: 0x2767a81 (stale) CRC-32 checksum value for chunk [0, 0x1a487): Mirror 1: 0x2767a81 Mirror 15: 0x2767a81 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a487: skipped 000360 1788836197.242747 mapread 0x00e719 thru 0x016313 (0x7bfb bytes) 000361 1788836197.336835 punch from 0x014a8a to 0x01586e (0xde5 bytes) 000362 1788836197.441735 MIRROR_RESYNC 8 mirrors 000363 1788836200.017050 punch from 0x00e8fd to 0x01441c (0x5b20 bytes) 000364 1788836200.125653 read 0x01883d thru 0x01a486 (0x1c4a bytes) 000365 1788836200.140488 trunc from 0x01a487 to 0x00d453 (0xd033 bytes) 000368 1788836200.185585 trunc from 0x00d454 to 0x00e010 (0xbbd bytes) 000370 1788836200.209249 write 0x0191f3 thru 0x01c434 (0x3242 bytes) 000372 1788836200.212291 write 0x01f1f4 thru 0x0240fe (0x4f0b bytes) 000373 1788836200.213723 punch from 0x005382 to 0x013fe6 (0xec65 bytes) 000374 1788836200.227102 punch from 0x017bfa to 0x0240fe (0xc505 bytes) 000377 1788836200.236892 mapread 0x012569 thru 0x0180a6 (0x5b3e bytes) 000378 1788836200.262283 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x240ff): Mirror 7: 0xf9f16e67 (stale) Mirror 11: 0xf9f16e67 (stale) Mirror 12: 0xf9f16e67 (stale) Mirror 13: 0xf9f16e67 (stale) Mirror 14: 0xf9f16e67 (stale) Mirror 15: 0xf9f16e67 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x240ff): Mirror 7: 0xf9f16e67 (stale) Mirror 11: 0xf9f16e67 (stale) Mirror 12: 0xf9f16e67 (stale) Mirror 13: 0xf9f16e67 (stale) Mirror 14: 0xf9f16e67 (stale) Mirror 15: 0xf9f16e67 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x240ff: skipped 000379 1788836200.970075 mapwrite 0x006297 thru 0x008a0f (0x2779 bytes) 000382 1788836201.034448 mapwrite 0x0183e8 thru 0x024755 (0xc36e bytes) 000383 1788836201.146631 MIRROR_RESYNC 7 mirrors 000384 1788836204.913677 read 0x00ed2c thru 0x010cc9 (0x1f9e bytes) 000385 1788836204.918755 mapwrite 0x010458 thru 0x010b05 (0x6ae bytes) 000386 1788836205.046124 punch from 0x01d720 to 0x01df24 (0x805 bytes) 000388 1788836205.052098 punch from 0x001730 to 0x005068 (0x3939 bytes) 000389 1788836205.057272 punch from 0x01230b to 0x0168cc (0x45c2 bytes) 000390 1788836205.061899 mapread 0x009dc5 thru 0x00c9b4 (0x2bf0 bytes) 000391 1788836205.075679 mapwrite 0x010bd1 thru 0x017594 (0x69c4 bytes) 000392 1788836205.130907 trunc from 0x024756 to 0x02ed73 (0xa61e bytes) 000393 1788836205.159993 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ed74): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ed74): Mirror 7: 0x24a76f68 (stale) Mirror 11: 0x24a76f68 (stale) Mirror 12: 0x24a76f68 (stale) Mirror 13: 0x24a76f68 (stale) Mirror 14: 0x24a76f68 (stale) Mirror 15: 0x24a76f68 (stale) CRC-32 checksum value for chunk [0, 0x2ed74): Mirror 1: 0xe6831220 Mirror 16: 0xe6831220 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ed74: skipped 000394 1788836206.663294 read 0x018037 thru 0x01e9f5 (0x69bf bytes) 000397 1788836206.694476 mapread 0x005c3d thru 0x00eda6 (0x916a bytes) 000398 1788836206.726399 write 0x03c28f thru 0x03ffff (0x3d71 bytes) 000400 1788836206.786799 write 0x035091 thru 0x03bbe0 (0x6b50 bytes) 000402 1788836206.802532 mapwrite 0x02ee31 thru 0x03de64 (0xf034 bytes) 000403 1788836206.876406 mapread 0x02fe30 thru 0x030808 (0x9d9 bytes) 000405 1788836206.878244 punch from 0x020459 to 0x02f37b (0xef23 bytes) 000406 1788836206.895728 mapwrite 0x015904 thru 0x022261 (0xc95e bytes) 000410 1788836207.058648 read 0x031a40 thru 0x03ffff (0xe5c0 bytes) 000413 1788836207.063845 trunc from 0x040000 to 0x017c8c (0x28373 bytes) 000415 1788836207.112336 read 0x0021ac thru 0x01055a (0xe3af bytes) 000416 1788836207.129084 trunc from 0x017c8d to 0x000a49 (0x17243 bytes) 000417 1788836207.148768 mapread 0x000424 thru 0x000a49 (0x626 bytes) 000418 1788836207.150847 mapread 0x000495 thru 0x000a49 (0x5b5 bytes) 000419 1788836207.152130 trunc from 0x000a4a to 0x015dfe (0x153b5 bytes) 000420 1788836207.170201 write 0x002698 thru 0x00ac1b (0x8584 bytes) 000421 1788836207.179625 trunc from 0x015dff to 0x037a26 (0x21c28 bytes) 000422 1788836207.193018 write 0x036a5d thru 0x03ffff (0x95a3 bytes) 000424 1788836207.205881 punch from 0x03a899 to 0x03ffff (0x5767 bytes) 000426 1788836207.225557 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x7e6f0867 (stale) Mirror 11: 0x7e6f0867 (stale) Mirror 12: 0x7e6f0867 (stale) Mirror 13: 0x7e6f0867 (stale) Mirror 14: 0x7e6f0867 (stale) Mirror 15: 0x7e6f0867 (stale) Mirror 16: 0xcf14afb3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf7b6480d Mirror 17: 0xf7b6480d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000427 1788836208.307415 mapread 0x02b4d7 thru 0x036de2 (0xb90c bytes) 000428 1788836208.360724 mapwrite 0x003c32 thru 0x00e534 (0xa903 bytes) 000430 1788836208.607057 read 0x0152d9 thru 0x01d2a3 (0x7fcb bytes) 000434 1788836208.620277 mapread 0x020224 thru 0x020be4 (0x9c1 bytes) 000435 1788836208.631801 trunc from 0x040000 to 0x01c7d6 (0x23829 bytes) 000436 1788836208.683901 read 0x0027ed thru 0x009ba3 (0x73b7 bytes) 000439 1788836208.687551 read 0x007d2c thru 0x012a26 (0xacfb bytes) 000440 1788836208.700210 punch from 0x00c7b4 to 0x0189b4 (0xc201 bytes) 000442 1788836208.707379 mapwrite 0x01199c thru 0x01394d (0x1fb2 bytes) 000444 1788836208.772023 write 0x013d05 thru 0x01b226 (0x7522 bytes) 000448 1788836208.778989 punch from 0x0190db to 0x01c7d6 (0x36fc bytes) 000449 1788836208.799350 punch from 0x01ab46 to 0x01c7d6 (0x1c91 bytes) 000450 1788836208.804939 punch from 0x00dd33 to 0x01ba94 (0xdd62 bytes) 000453 1788836208.815111 mapread 0x00e6c3 thru 0x00f705 (0x1043 bytes) 000454 1788836208.831536 mapread 0x00f866 thru 0x01c7d6 (0xcf71 bytes) 000455 1788836208.901299 write 0x0282e4 thru 0x030964 (0x8681 bytes) 000456 1788836208.903054 mapwrite 0x037b6a thru 0x0394b1 (0x1948 bytes) 000458 1788836208.992988 write 0x03035a thru 0x0362c0 (0x5f67 bytes) 000459 1788836208.994597 write 0x00cc7d thru 0x01926b (0xc5ef bytes) 000460 1788836209.001500 write 0x0372d1 thru 0x037fa1 (0xcd1 bytes) 000461 1788836209.018900 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x394b2): Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x394b2): Mirror 7: 0xc4d36f5b (stale) Mirror 11: 0xc4d36f5b (stale) Mirror 12: 0xc4d36f5b (stale) Mirror 13: 0xc4d36f5b (stale) Mirror 14: 0xc4d36f5b (stale) Mirror 15: 0xc4d36f5b (stale) Mirror 16: 0x619be785 (stale) Mirror 17: 0xbcc5dbff (stale) CRC-32 checksum value for chunk [0, 0x394b2): Mirror 1: 0xb5aba929 Mirror 18: 0xb5aba929 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x394b2: skipped 000463 1788836210.482877 write 0x01b36d thru 0x025754 (0xa3e8 bytes) 000464 1788836210.636231 trunc from 0x0394b2 to 0x005286 (0x3422b bytes) 000465 1788836210.662579 read 0x00052b thru 0x005286 (0x4d5c bytes) 000466 1788836210.674526 write 0x00511b thru 0x00e333 (0x9219 bytes) 000467 1788836210.676298 punch from 0x00751c to 0x00e333 (0x6e18 bytes) 000468 1788836210.696881 punch from 0x00b400 to 0x00e333 (0x2f34 bytes) 000472 1788836210.708556 write 0x01b95c thru 0x0246d6 (0x8d7b bytes) 000473 1788836210.711727 trunc from 0x0246d7 to 0x02e34a (0x9c74 bytes) 000474 1788836210.759734 mapwrite 0x038702 thru 0x03eb97 (0x6496 bytes) 000475 1788836210.889338 write 0x02b06d thru 0x02f80b (0x479f bytes) 000478 1788836210.905268 MIRROR_RESYNC 10 mirrors 000480 1788836217.656569 read 0x02e7af thru 0x03a70d (0xbf5f bytes) 000481 1788836217.665856 mapwrite 0x01e27c thru 0x026fbf (0x8d44 bytes) 000482 1788836217.858634 mapwrite 0x027ac0 thru 0x02d08b (0x55cc bytes) 000486 1788836217.939019 punch from 0x0289f6 to 0x029a79 (0x1084 bytes) 000487 1788836217.948759 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb98): Mirror 11: 0x75f363 (stale) Mirror 12: 0x75f363 (stale) Mirror 13: 0x75f363 (stale) Mirror 14: 0x75f363 (stale) Mirror 15: 0x75f363 (stale) Mirror 16: 0x75f363 (stale) Mirror 17: 0x75f363 (stale) Mirror 18: 0x75f363 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb98): Mirror 11: 0x75f363 (stale) Mirror 12: 0x75f363 (stale) Mirror 13: 0x75f363 (stale) Mirror 14: 0x75f363 (stale) Mirror 15: 0x75f363 (stale) Mirror 16: 0x75f363 (stale) Mirror 17: 0x75f363 (stale) Mirror 18: 0x75f363 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb98: skipped 000488 1788836218.769573 trunc from 0x03eb98 to 0x0286ec (0x164ab bytes) 000489 1788836218.793018 trunc from 0x0286ed to 0x022e3f (0x58ad bytes) 000490 1788836218.818382 mapread 0x01c9c8 thru 0x022e3f (0x6478 bytes) 000491 1788836218.843719 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22e40): Mirror 11: 0xdb94ec85 (stale) Mirror 12: 0xdb94ec85 (stale) Mirror 13: 0xdb94ec85 (stale) Mirror 14: 0xdb94ec85 (stale) Mirror 15: 0xdb94ec85 (stale) Mirror 16: 0xdb94ec85 (stale) Mirror 18: 0xdb94ec85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22e40): Mirror 11: 0xdb94ec85 (stale) Mirror 12: 0xdb94ec85 (stale) Mirror 13: 0xdb94ec85 (stale) Mirror 14: 0xdb94ec85 (stale) Mirror 15: 0xdb94ec85 (stale) Mirror 16: 0xdb94ec85 (stale) Mirror 18: 0xdb94ec85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22e40: skipped 000492 1788836219.399042 mapwrite 0x0147b9 thru 0x018113 (0x395b bytes) 000493 1788836219.490856 write 0x01ee9d thru 0x02bf9d (0xd101 bytes) 000494 1788836219.493883 punch from 0x00132f to 0x003215 (0x1ee7 bytes) 000495 1788836219.512080 read 0x01a3b6 thru 0x01b224 (0xe6f bytes) 000496 1788836219.513753 write 0x01cb6b thru 0x02197f (0x4e15 bytes) 000497 1788836219.516766 write 0x0217d1 thru 0x030dca (0xf5fa bytes) 000498 1788836219.521027 read 0x0133a2 thru 0x021fdb (0xec3a bytes) 000501 1788836219.524489 punch from 0x004686 to 0x008511 (0x3e8c bytes) 000504 1788836219.543823 trunc from 0x030dcb to 0x039a0c (0x8c42 bytes) 000505 1788836219.562977 mapwrite 0x01a4f4 thru 0x0233de (0x8eeb bytes) 000506 1788836219.679540 write 0x01958e thru 0x023792 (0xa205 bytes) 000508 1788836219.689363 punch from 0x00fa99 to 0x010b21 (0x1089 bytes) 000509 1788836219.710034 mapwrite 0x02fb20 thru 0x03a967 (0xae48 bytes) 000510 1788836219.809336 trunc from 0x03a968 to 0x01329b (0x276cc bytes) 000512 1788836219.833337 trunc from 0x01329c to 0x0197d6 (0x653b bytes) 000513 1788836219.855403 read 0x011e5f thru 0x0197d6 (0x7978 bytes) 000514 1788836219.872686 punch from 0x010400 to 0x0197d6 (0x93d7 bytes) 000515 1788836219.878722 punch from 0x01654e to 0x0172d5 (0xd88 bytes) 000517 1788836219.890390 MIRROR_RESYNC 8 mirrors 000520 1788836224.181021 read 0x019156 thru 0x0197d6 (0x681 bytes) 000522 1788836224.186889 write 0x02bb1d thru 0x0335bf (0x7aa3 bytes) 000523 1788836224.283830 write 0x032dc8 thru 0x036305 (0x353e bytes) 000524 1788836224.286541 punch from 0x0164f1 to 0x0232e8 (0xcdf8 bytes) 000525 1788836224.308672 punch from 0x02249f to 0x02c99a (0xa4fc bytes) 000528 1788836224.323324 MIRROR_RESYNC 8 mirrors 000529 1788836229.049393 mapwrite 0x003654 thru 0x01314f (0xfafc bytes) 000530 1788836229.411291 mapwrite 0x0299b5 thru 0x0344fa (0xab46 bytes) 000532 1788836229.564463 punch from 0x035454 to 0x036305 (0xeb2 bytes) 000533 1788836229.582696 mapread 0x014bdf thru 0x01da57 (0x8e79 bytes) 000535 1788836229.596527 write 0x03fcdf thru 0x03ffff (0x321 bytes) 000536 1788836229.597811 mapread 0x036ea6 thru 0x03ffff (0x915a bytes) 000537 1788836229.631684 punch from 0x01a398 to 0x01a867 (0x4d0 bytes) 000538 1788836229.660812 write 0x01de25 thru 0x01f8b8 (0x1a94 bytes) 000539 1788836229.665673 mapread 0x03eb2d thru 0x03ffff (0x14d3 bytes) 000540 1788836229.668373 mapwrite 0x021a25 thru 0x030827 (0xee03 bytes) 000541 1788836229.774326 mapwrite 0x012df9 thru 0x01d820 (0xaa28 bytes) 000543 1788836229.833269 mapwrite 0x024da5 thru 0x028e28 (0x4084 bytes) 000544 1788836229.881112 punch from 0x007626 to 0x0089c0 (0x139b bytes) 000545 1788836229.886559 read 0x00b393 thru 0x00bcf3 (0x961 bytes) 000549 1788836229.889841 punch from 0x00726a to 0x0078da (0x671 bytes) 000550 1788836229.894541 mapwrite 0x03c785 thru 0x03ffff (0x387b bytes) 000551 1788836229.984971 trunc from 0x040000 to 0x0055b7 (0x3aa48 bytes) 000552 1788836230.036628 read 0x001e8b thru 0x0055b7 (0x372d bytes) 000554 1788836230.040490 mapread 0x0033c4 thru 0x0055b7 (0x21f4 bytes) 000555 1788836230.046647 MIRROR_RESYNC 8 mirrors 000556 1788836232.761269 read 0x000001 thru 0x0055b7 (0x55b7 bytes) 000557 1788836232.781015 mapwrite 0x03ff56 thru 0x03ffff (0xaa bytes) 000558 1788836232.954899 mapread 0x007837 thru 0x009869 (0x2033 bytes) 000559 1788836232.979607 trunc from 0x040000 to 0x035d14 (0xa2eb bytes) 000561 1788836233.009063 punch from 0x014347 to 0x014385 (0x3f bytes) 000562 1788836233.014944 read 0x000377 thru 0x00a1be (0x9e48 bytes) 000563 1788836233.036642 trunc from 0x035d15 to 0x014aaa (0x2126a bytes) 000564 1788836233.061764 write 0x01fc6c thru 0x0264da (0x686f bytes) 000566 1788836233.064790 write 0x03c5f4 thru 0x03ffff (0x3a0c bytes) 000567 1788836233.068332 punch from 0x033aa7 to 0x03db84 (0xa0de bytes) 000570 1788836233.095870 read 0x01f875 thru 0x02682f (0x6fbb bytes) 000572 1788836233.100390 read 0x01cf85 thru 0x02225f (0x52db bytes) 000573 1788836233.114967 read 0x039e69 thru 0x03d66f (0x3807 bytes) 000574 1788836233.115589 trunc from 0x040000 to 0x032f80 (0xd07f bytes) 000575 1788836233.142745 punch from 0x01bcf6 to 0x01f2c6 (0x35d1 bytes) 000576 1788836233.150145 mapread 0x029824 thru 0x032f80 (0x975d bytes) 000579 1788836233.153064 trunc from 0x032f81 to 0x03ea25 (0xbaa5 bytes) 000581 1788836233.174883 trunc from 0x03ea26 to 0x017828 (0x271fd bytes) 000582 1788836233.199181 read 0x008c3d thru 0x00a9db (0x1d9f bytes) 000583 1788836233.200694 mapread 0x012800 thru 0x017828 (0x5029 bytes) 000584 1788836233.215754 MIRROR_RESYNC 8 mirrors 000585 1788836235.636085 punch from 0x000247 to 0x009dc3 (0x9b7d bytes) 000587 1788836235.772683 mapread 0x0094be thru 0x016de2 (0xd925 bytes) 000588 1788836235.800363 trunc from 0x017829 to 0x03d3c0 (0x25b98 bytes) 000589 1788836235.824602 trunc from 0x03d3c1 to 0x03d6eb (0x32b bytes) 000590 1788836235.868340 trunc from 0x03d6ec to 0x002578 (0x3b173 bytes) 000591 1788836235.905680 mapwrite 0x00a168 thru 0x00cf00 (0x2d99 bytes) 000593 1788836236.076417 read 0x005a36 thru 0x00c05b (0x6626 bytes) 000594 1788836236.076995 write 0x021938 thru 0x026e4e (0x5517 bytes) 000595 1788836236.084882 MIRROR_RESYNC 8 mirrors 000596 1788836239.694558 mapread 0x024c9e thru 0x026e4e (0x21b1 bytes) 000597 1788836239.721885 trunc from 0x026e4f to 0x008ecf (0x1df7f bytes) 000599 1788836239.917929 read 0x001f16 thru 0x008ecf (0x6fba bytes) 000600 1788836239.928331 read 0x00549d thru 0x008ecf (0x3a33 bytes) 000601 1788836239.929048 trunc from 0x008ed0 to 0x01d987 (0x14ab8 bytes) 000602 1788836239.963093 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1d988): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d988): Mirror 11: 0xa68918ff (stale) Mirror 12: 0xa68918ff (stale) Mirror 13: 0xa68918ff (stale) Mirror 14: 0xa68918ff (stale) Mirror 15: 0xa68918ff (stale) Mirror 16: 0xa68918ff (stale) Mirror 18: 0xa68918ff (stale) CRC-32 checksum value for chunk [0, 0x1d988): Mirror 1: 0x85b6fb6b Mirror 19: 0x85b6fb6b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d988: skipped 000606 1788836241.723710 punch from 0x01b8c6 to 0x01d987 (0x20c2 bytes) 000607 1788836241.824140 mapread 0x00290d thru 0x009928 (0x701c bytes) 000608 1788836241.843449 mapwrite 0x00ae52 thru 0x013035 (0x81e4 bytes) 000609 1788836241.907772 read 0x00ebb4 thru 0x01494e (0x5d9b bytes) 000610 1788836241.909578 write 0x01e0d8 thru 0x0262f3 (0x821c bytes) 000611 1788836241.912279 write 0x03b18c thru 0x03ffff (0x4e74 bytes) 000613 1788836241.914566 punch from 0x031aec to 0x0390e9 (0x75fe bytes) 000614 1788836241.927535 read 0x010aa0 thru 0x016e76 (0x63d7 bytes) 000615 1788836241.928533 mapread 0x00a2bf thru 0x019008 (0xed4a bytes) 000617 1788836241.930465 mapwrite 0x00ce93 thru 0x014667 (0x77d5 bytes) 000618 1788836241.995762 trunc from 0x040000 to 0x015e50 (0x2a1af bytes) 000620 1788836242.018215 read 0x00cbe1 thru 0x015e50 (0x9270 bytes) 000621 1788836242.020345 write 0x00a18c thru 0x00f7ed (0x5662 bytes) 000622 1788836242.022110 mapwrite 0x0390a1 thru 0x03ffff (0x6f5f bytes) 000623 1788836242.115328 write 0x02d3e4 thru 0x02f492 (0x20af bytes) 000624 1788836242.116330 read 0x02a16d thru 0x036b9b (0xca2f bytes) 000626 1788836242.119380 trunc from 0x040000 to 0x034e1c (0xb1e3 bytes) 000627 1788836242.148329 read 0x00dd1a thru 0x019cc8 (0xbfaf bytes) 000630 1788836242.150915 write 0x004164 thru 0x012523 (0xe3c0 bytes) 000633 1788836242.158984 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34e1d): Mirror 11: 0xcef703a6 (stale) Mirror 12: 0xcef703a6 (stale) Mirror 13: 0xcef703a6 (stale) Mirror 15: 0xcef703a6 (stale) Mirror 16: 0xcef703a6 (stale) Mirror 18: 0xcef703a6 (stale) Mirror 19: 0xdd7209de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e1d): Mirror 11: 0xcef703a6 (stale) Mirror 12: 0xcef703a6 (stale) Mirror 13: 0xcef703a6 (stale) Mirror 15: 0xcef703a6 (stale) Mirror 16: 0xcef703a6 (stale) Mirror 18: 0xcef703a6 (stale) Mirror 19: 0xdd7209de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e1d: skipped 000634 1788836242.625466 read 0x0339cd thru 0x034e1c (0x1450 bytes) 000637 1788836242.632261 mapwrite 0x005efb thru 0x01173e (0xb844 bytes) 000638 1788836242.717807 read 0x00beb4 thru 0x00de21 (0x1f6e bytes) 000639 1788836242.718468 mapwrite 0x0091d2 thru 0x012bd7 (0x9a06 bytes) 000640 1788836242.770094 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x34e1d): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e1d): Mirror 11: 0xcef703a6 (stale) Mirror 12: 0xcef703a6 (stale) Mirror 13: 0xcef703a6 (stale) Mirror 15: 0xcef703a6 (stale) Mirror 16: 0xcef703a6 (stale) Mirror 18: 0xcef703a6 (stale) Mirror 19: 0xdd7209de (stale) CRC-32 checksum value for chunk [0, 0x34e1d): Mirror 1: 0x9c63dd2c Mirror 20: 0x9c63dd2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e1d: skipped 000641 1788836244.087452 write 0x037a10 thru 0x03ffff (0x85f0 bytes) 000642 1788836244.232372 mapread 0x03c3ed thru 0x03ffff (0x3c13 bytes) 000645 1788836244.235460 write 0x00420d thru 0x0056ad (0x14a1 bytes) 000646 1788836244.252104 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xbc0837a8 (stale) Mirror 13: 0xbc0837a8 (stale) Mirror 15: 0xbc0837a8 (stale) Mirror 16: 0xbc0837a8 (stale) Mirror 18: 0xbc0837a8 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xbc14ea80 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 11: 0xbc0837a8 (stale) Mirror 13: 0xbc0837a8 (stale) Mirror 15: 0xbc0837a8 (stale) Mirror 16: 0xbc0837a8 (stale) Mirror 18: 0xbc0837a8 (stale) Mirror 19: 0xe20eea22 (stale) Mirror 20: 0xbc14ea80 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000650 1788836244.870139 mapread 0x016750 thru 0x023d20 (0xd5d1 bytes) 000653 1788836244.899141 mapread 0x00a476 thru 0x00d62d (0x31b8 bytes) 000654 1788836244.906589 write 0x026bdd thru 0x032a97 (0xbebb bytes) 000656 1788836244.912464 mapread 0x02cd8c thru 0x033998 (0x6c0d bytes) 000657 1788836244.920766 punch from 0x016a9b to 0x024684 (0xdbea bytes) 000658 1788836244.932221 mapread 0x034e06 thru 0x03a472 (0x566d bytes) 000659 1788836244.933115 punch from 0x0343eb to 0x03f080 (0xac96 bytes) 000660 1788836244.938123 punch from 0x0019c5 to 0x003300 (0x193c bytes) 000663 1788836244.944097 punch from 0x025eaf to 0x02dc1a (0x7d6c bytes) 000664 1788836244.948848 mapread 0x03ad07 thru 0x03ffff (0x52f9 bytes) 000665 1788836244.957126 punch from 0x015373 to 0x01bfcd (0x6c5b bytes) 000667 1788836244.961675 trunc from 0x040000 to 0x00599c (0x3a663 bytes) 000668 1788836244.976357 punch from 0x004fd8 to 0x00599c (0x9c5 bytes) 000669 1788836244.985997 MIRROR_RESYNC 8 mirrors 000671 1788836247.264437 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000672 1788836247.701879 MIRROR_EXTEND to 8 mirrors 000676 1788836248.905378 mapwrite 0x01929b thru 0x02457a (0xb2e0 bytes) 000677 1788836249.182310 trunc from 0x02457b to 0x0376a7 (0x1312d bytes) 000678 1788836249.232769 read 0x0182fb thru 0x01af96 (0x2c9c bytes) 000679 1788836249.233486 mapwrite 0x01a96b thru 0x0256d0 (0xad66 bytes) 000680 1788836249.344996 MIRROR_RESYNC 8 mirrors 000681 1788836253.225559 write 0x0223bc thru 0x025432 (0x3077 bytes) 000682 1788836253.341301 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x376a8): Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376a8): Mirror 11: 0x7867ca92 (stale) Mirror 13: 0x7867ca92 (stale) Mirror 15: 0x7867ca92 (stale) Mirror 18: 0x7867ca92 (stale) Mirror 19: 0x7867ca92 (stale) Mirror 20: 0x7867ca92 (stale) Mirror 21: 0x7867ca92 (stale) CRC-32 checksum value for chunk [0, 0x376a8): Mirror 1: 0xbc70bd51 Mirror 22: 0xbc70bd51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376a8: skipped 000684 1788836254.833561 write 0x01ab43 thru 0x01da1a (0x2ed8 bytes) 000685 1788836254.925195 read 0x00b943 thru 0x010262 (0x4920 bytes) 000687 1788836254.940047 write 0x03de8d thru 0x03ffff (0x2173 bytes) 000688 1788836254.941407 trunc from 0x040000 to 0x028db6 (0x17249 bytes) 000690 1788836254.961503 trunc from 0x028db7 to 0x039abf (0x10d09 bytes) 000693 1788836254.984986 MIRROR_RESYNC 9 mirrors 000694 1788836259.027969 read 0x037aa7 thru 0x039abf (0x2019 bytes) 000695 1788836259.040383 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000696 1788836259.671041 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) 000698 1788836260.183838 trunc from 0x039ac0 to 0x002334 (0x3778b bytes) 000700 1788836260.316986 punch from 0x001a5a to 0x002334 (0x8db bytes) 000703 1788836260.328913 mapwrite 0x03530e thru 0x038baa (0x389d bytes) 000704 1788836260.558908 MIRROR_RESYNC 7 mirrors 000705 1788836263.013642 mapwrite 0x010e36 thru 0x01e74c (0xd917 bytes) 000706 1788836263.519943 trunc from 0x038bab to 0x017afa (0x210b0 bytes) 000707 1788836263.554825 read 0x015200 thru 0x017afa (0x28fb bytes) 000709 1788836263.556033 read 0x012976 thru 0x017afa (0x5185 bytes) 000710 1788836263.556584 write 0x03b5ba thru 0x03ffff (0x4a46 bytes) 000711 1788836263.557890 mapread 0x032c02 thru 0x03de23 (0xb222 bytes) 000714 1788836263.571486 trunc from 0x040000 to 0x02353f (0x1cac0 bytes) 000716 1788836263.606210 mapread 0x0210a0 thru 0x02353f (0x24a0 bytes) 000717 1788836263.607586 punch from 0x0141b1 to 0x01a45c (0x62ac bytes) 000718 1788836263.615265 mapwrite 0x007037 thru 0x008bba (0x1b84 bytes) 000719 1788836263.809795 punch from 0x0075e9 to 0x00f377 (0x7d8f bytes) 000720 1788836263.818542 read 0x00cc49 thru 0x010eb9 (0x4271 bytes) 000721 1788836263.834977 MIRROR_RESYNC 7 mirrors 000723 1788836267.442223 write 0x03f0a2 thru 0x03ffff (0xf5e bytes) 000724 1788836267.638792 mapread 0x015aa0 thru 0x017658 (0x1bb9 bytes) 000725 1788836267.674914 mapread 0x00769f thru 0x00bb97 (0x44f9 bytes) 000727 1788836267.693929 mapread 0x0255e9 thru 0x028a79 (0x3491 bytes) 000728 1788836267.714040 write 0x01004e thru 0x020026 (0xffd9 bytes) 000729 1788836267.717904 trunc from 0x040000 to 0x01fe05 (0x201fa bytes) 000730 1788836267.757914 read 0x011cf6 thru 0x019409 (0x7714 bytes) 000731 1788836267.759639 mapread 0x0168da thru 0x01c0cb (0x57f2 bytes) 000733 1788836267.764307 read 0x00f17f thru 0x0162a7 (0x7129 bytes) 000736 1788836267.766887 punch from 0x01477e to 0x01ec7e (0xa501 bytes) 000743 1788836267.816000 write 0x02939c thru 0x034d0b (0xb970 bytes) 000744 1788836267.822045 mapwrite 0x00dc40 thru 0x00ebcc (0xf8d bytes) 000746 1788836267.949238 MIRROR_RESYNC 7 mirrors 000747 1788836274.655970 punch from 0x00959e to 0x00dd0d (0x4770 bytes) 000753 1788836274.929760 mapwrite 0x00c6ed thru 0x00fa80 (0x3394 bytes) 000754 1788836275.168680 MIRROR_RESYNC 7 mirrors 000755 1788836279.501200 mapwrite 0x01af47 thru 0x029ee4 (0xef9e bytes) 000756 1788836279.756078 mapread 0x02e9ca thru 0x034cf9 (0x6330 bytes) 000757 1788836279.767442 mapwrite 0x038df6 thru 0x03ffff (0x720a bytes) 000759 1788836279.977961 trunc from 0x040000 to 0x021e75 (0x1e18a bytes) 000760 1788836280.008884 trunc from 0x021e76 to 0x001dce (0x200a7 bytes) 000761 1788836280.032014 mapwrite 0x02c37d thru 0x0308c0 (0x4544 bytes) 000763 1788836280.189867 read 0x013ea5 thru 0x0169ff (0x2b5b bytes) 000764 1788836280.198740 write 0x019551 thru 0x029211 (0xfcc1 bytes) 000765 1788836280.209536 mapread 0x010fc0 thru 0x020d80 (0xfdc1 bytes) 000768 1788836280.226409 punch from 0x00958f to 0x00991b (0x38d bytes) 000769 1788836280.247858 write 0x03297f thru 0x03b1e8 (0x886a bytes) 000772 1788836280.252587 read 0x019481 thru 0x029337 (0xfeb7 bytes) 000774 1788836280.256004 write 0x034f5b thru 0x03c9bf (0x7a65 bytes) 000775 1788836280.269605 MIRROR_RESYNC 7 mirrors 000776 1788836283.733518 mapwrite 0x005ed3 thru 0x013545 (0xd673 bytes) 000778 1788836283.953301 read 0x0086ba thru 0x0138db (0xb222 bytes) 000781 1788836283.954951 trunc from 0x03c9c0 to 0x015586 (0x27439 bytes) 000783 1788836283.998752 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15587): Mirror 13: 0x9c9cc5cb (stale) Mirror 18: 0x9c9cc5cb (stale) Mirror 20: 0x9c9cc5cb (stale) Mirror 21: 0x9c9cc5cb (stale) Mirror 22: 0x9c9cc5cb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15587): Mirror 13: 0x9c9cc5cb (stale) Mirror 18: 0x9c9cc5cb (stale) Mirror 20: 0x9c9cc5cb (stale) Mirror 21: 0x9c9cc5cb (stale) Mirror 22: 0x9c9cc5cb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15587: skipped 000784 1788836284.538570 read 0x00ed35 thru 0x015586 (0x6852 bytes) 000785 1788836284.562837 mapread 0x010769 thru 0x015252 (0x4aea bytes) 000787 1788836284.568044 read 0x00eb4e thru 0x013e26 (0x52d9 bytes) 000788 1788836284.570965 trunc from 0x015587 to 0x014ba1 (0x9e5 bytes) 000790 1788836284.589226 mapread 0x005fc4 thru 0x01198e (0xb9cb bytes) 000791 1788836284.600797 mapread 0x00f8f4 thru 0x014ba1 (0x52ae bytes) 000792 1788836284.602633 trunc from 0x014ba2 to 0x020bf4 (0xc053 bytes) 000794 1788836284.628905 mapread 0x01008f thru 0x017894 (0x7806 bytes) 000795 1788836284.638957 mapwrite 0x015fb4 thru 0x01f02d (0x907a bytes) 000796 1788836284.699635 read 0x01cd24 thru 0x020bf4 (0x3ed1 bytes) 000797 1788836284.700279 read 0x01ffc0 thru 0x020bf4 (0xc35 bytes) 000799 1788836284.701233 trunc from 0x020bf5 to 0x02b8e1 (0xaced bytes) 000800 1788836284.728095 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b8e2): Mirror 18: 0xfa52ee29 (stale) Mirror 20: 0xfa52ee29 (stale) Mirror 21: 0xfa52ee29 (stale) Mirror 22: 0xfa52ee29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b8e2): Mirror 18: 0xfa52ee29 (stale) Mirror 20: 0xfa52ee29 (stale) Mirror 21: 0xfa52ee29 (stale) Mirror 22: 0xfa52ee29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b8e2: skipped 000801 1788836285.279536 punch from 0x0097fb to 0x01518f (0xb995 bytes) 000804 1788836285.289399 punch from 0x00fba2 to 0x0114e1 (0x1940 bytes) 000808 1788836285.300040 write 0x028e39 thru 0x037b3a (0xed02 bytes) 000809 1788836285.311372 mapread 0x00993f thru 0x0192ea (0xf9ac bytes) 000810 1788836285.341719 write 0x01c4e9 thru 0x021ae2 (0x55fa bytes) 000811 1788836285.347901 trunc from 0x037b3b to 0x01856b (0x1f5cf bytes) 000813 1788836285.380396 punch from 0x003e69 to 0x0120e3 (0xe27b bytes) 000815 1788836285.390194 read 0x0107f3 thru 0x0156b6 (0x4ec4 bytes) 000816 1788836285.410779 MIRROR_RESYNC 5 mirrors 000818 1788836287.461106 write 0x02bc67 thru 0x033fe1 (0x837b bytes) 000820 1788836287.531259 write 0x0295aa thru 0x02d17b (0x3bd2 bytes) 000822 1788836287.543904 mapwrite 0x02800d thru 0x032c2b (0xac1f bytes) 000823 1788836287.603964 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 18: 0x51df0905 (stale) Mirror 20: 0x51df0905 (stale) Mirror 21: 0x51df0905 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fe2): Mirror 18: 0x51df0905 (stale) Mirror 20: 0x51df0905 (stale) Mirror 21: 0x51df0905 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fe2: skipped 000824 1788836287.954825 MIRROR_RESYNC 4 mirrors 000825 1788836290.017048 read 0x00f3c1 thru 0x00f95d (0x59d bytes) 000826 1788836290.021744 read 0x01f12d thru 0x02d9de (0xe8b2 bytes) 000828 1788836290.035925 punch from 0x026e04 to 0x02a5b9 (0x37b6 bytes) 000831 1788836290.116033 read 0x02ca78 thru 0x033fe1 (0x756a bytes) 000832 1788836290.131384 write 0x014765 thru 0x01ea80 (0xa31c bytes) 000835 1788836290.135340 mapread 0x018012 thru 0x01c2f8 (0x42e7 bytes) 000836 1788836290.136434 trunc from 0x033fe2 to 0x030468 (0x3b79 bytes) 000837 1788836290.161301 mapread 0x01426a thru 0x017688 (0x341f bytes) 000838 1788836290.166806 MIRROR_RESYNC 4 mirrors 000841 1788836292.424629 mapread 0x01ecd4 thru 0x026942 (0x7c6f bytes) 000842 1788836292.494915 trunc from 0x030469 to 0x0213df (0xf089 bytes) 000843 1788836292.651209 read 0x0006f5 thru 0x00f469 (0xed75 bytes) 000844 1788836292.664613 mapwrite 0x00b65d thru 0x011083 (0x5a27 bytes) 000845 1788836292.808803 MIRROR_RESYNC 4 mirrors 000846 1788836295.021043 MIRROR_EXTEND to 5 mirrors 000848 1788836295.992621 write 0x01c2ec thru 0x029e61 (0xdb76 bytes) 000850 1788836296.110538 mapwrite 0x021fd7 thru 0x02f6fc (0xd726 bytes) 000851 1788836296.226217 trunc from 0x02f6fd to 0x02fdaa (0x6ae bytes) 000853 1788836296.251895 punch from 0x00afe0 to 0x011e6b (0x6e8c bytes) 000856 1788836296.260573 write 0x03d3c9 thru 0x03ffff (0x2c37 bytes) 000859 1788836296.263847 mapwrite 0x022947 thru 0x023f1f (0x15d9 bytes) 000860 1788836296.328540 mapread 0x00db11 thru 0x01b062 (0xd552 bytes) 000861 1788836296.350500 read 0x03adcb thru 0x03cf96 (0x21cc bytes) 000866 1788836296.363007 mapwrite 0x024d56 thru 0x0276f9 (0x29a4 bytes) 000867 1788836296.416076 punch from 0x02546f to 0x02f49c (0xa02e bytes) 000868 1788836296.422594 mapread 0x0268f0 thru 0x02a733 (0x3e44 bytes) 000869 1788836296.431909 punch from 0x014f21 to 0x023fac (0xf08c bytes) 000870 1788836296.437511 mapread 0x01d6fa thru 0x02497b (0x7282 bytes) 000871 1788836296.449449 mapwrite 0x01bbed thru 0x020995 (0x4da9 bytes) 000872 1788836296.515578 mapread 0x0372aa thru 0x03ffff (0x8d56 bytes) 000874 1788836296.527554 mapwrite 0x00cf5e thru 0x01b51f (0xe5c2 bytes) 000875 1788836296.600843 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x1c6b6d49 (stale) Mirror 21: 0x1c6b6d49 (stale) Mirror 22: 0x1c6b6d49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x1c6b6d49 (stale) Mirror 21: 0x1c6b6d49 (stale) Mirror 22: 0x1c6b6d49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000876 1788836297.003695 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x1c6b6d49 (stale) Mirror 21: 0x1c6b6d49 (stale) Mirror 22: 0x1c6b6d49 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1496dd81 Mirror 23: 0x1496dd81 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000877 1788836298.373841 mapwrite 0x011e6a thru 0x0213a4 (0xf53b bytes) 000878 1788836298.752842 mapwrite 0x014be4 thru 0x020088 (0xb4a5 bytes) 000879 1788836298.858754 write 0x02c1d4 thru 0x034876 (0x86a3 bytes) 000882 1788836298.895831 punch from 0x037266 to 0x03ffff (0x8d9a bytes) 000884 1788836298.928506 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x1c6b6d49 (stale) Mirror 21: 0x1c6b6d49 (stale) Mirror 22: 0x1c6b6d49 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x1c6b6d49 (stale) Mirror 21: 0x1c6b6d49 (stale) Mirror 22: 0x1c6b6d49 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000886 1788836299.455307 read 0x010e1e thru 0x01f488 (0xe66b bytes) 000887 1788836299.473335 write 0x035f11 thru 0x03e224 (0x8314 bytes) 000888 1788836299.494211 write 0x028771 thru 0x0328b3 (0xa143 bytes) 000889 1788836299.510437 mapread 0x005eb2 thru 0x011cb9 (0xbe08 bytes) 000890 1788836299.540262 mapwrite 0x01c47c thru 0x0242e6 (0x7e6b bytes) 000892 1788836299.643615 mapread 0x037c24 thru 0x03abc8 (0x2fa5 bytes) 000895 1788836299.651454 punch from 0x0314a1 to 0x03d202 (0xbd62 bytes) 000896 1788836299.662824 punch from 0x0206e0 to 0x02c366 (0xbc87 bytes) 000898 1788836299.679385 write 0x01438e thru 0x019ba1 (0x5814 bytes) 000899 1788836299.681729 mapread 0x002075 thru 0x00b207 (0x9193 bytes) 000900 1788836299.684373 punch from 0x019539 to 0x01c612 (0x30da bytes) 000901 1788836299.701984 read 0x01ac24 thru 0x02913f (0xe51c bytes) 000902 1788836299.717873 MIRROR_RESYNC 4 mirrors 000903 1788836302.532002 mapwrite 0x00d39b thru 0x0161fe (0x8e64 bytes) 000906 1788836302.792315 write 0x00a6f6 thru 0x0195d7 (0xeee2 bytes) 000908 1788836302.813906 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 20: 0x600989b9 (stale) Mirror 21: 0x600989b9 (stale) Mirror 22: 0x600989b9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x460fa997 Mirror 23: 0x460fa997 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000909 1788836304.831705 trunc from 0x040000 to 0x027cf9 (0x18306 bytes) 000910 1788836304.985016 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27cfa): Mirror 21: 0x7f845d2 (stale) Mirror 22: 0x7f845d2 (stale) Mirror 23: 0x8306d557 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27cfa): Mirror 21: 0x7f845d2 (stale) Mirror 22: 0x7f845d2 (stale) Mirror 23: 0x8306d557 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27cfa: skipped 000912 1788836305.494536 mapwrite 0x008721 thru 0x01752e (0xee0e bytes) 000913 1788836305.624429 punch from 0x00a5a7 to 0x0109e0 (0x643a bytes) 000915 1788836305.635214 trunc from 0x027cfa to 0x022451 (0x58a8 bytes) 000917 1788836305.659064 MIRROR_RESYNC 4 mirrors 000918 1788836308.052043 write 0x03b0d7 thru 0x03ffff (0x4f29 bytes) 000919 1788836308.147594 punch from 0x014b93 to 0x0216bf (0xcb2d bytes) 000920 1788836308.166377 read 0x01fdcc thru 0x029a15 (0x9c4a bytes) 000921 1788836308.176260 trunc from 0x040000 to 0x013b76 (0x2c489 bytes) 000922 1788836308.196612 trunc from 0x013b77 to 0x031a7d (0x1df07 bytes) 000923 1788836308.213056 punch from 0x00e65d to 0x012bdf (0x4583 bytes) 000925 1788836308.218649 punch from 0x02841e to 0x02b3e1 (0x2fc4 bytes) 000926 1788836308.224488 mapwrite 0x0277e0 thru 0x02b56a (0x3d8b bytes) 000927 1788836308.294090 read 0x02e98e thru 0x02fedb (0x154e bytes) 000930 1788836308.295721 write 0x02e824 thru 0x035a9c (0x7279 bytes) 000931 1788836308.303777 write 0x0371fd thru 0x03baf6 (0x48fa bytes) 000932 1788836308.309134 write 0x01ed0a thru 0x01f761 (0xa58 bytes) 000933 1788836308.333398 write 0x01aef4 thru 0x02970b (0xe818 bytes) 000934 1788836308.336026 punch from 0x004627 to 0x0048f3 (0x2cd bytes) 000935 1788836308.361195 read 0x019410 thru 0x028bb3 (0xf7a4 bytes) 000937 1788836308.364387 trunc from 0x03baf7 to 0x0062aa (0x3584c bytes) 000938 1788836308.388066 mapread 0x003727 thru 0x0062aa (0x2b84 bytes) 000939 1788836308.401119 trunc from 0x0062ab to 0x00c6ac (0x6402 bytes) 000940 1788836308.432549 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc6ad): Mirror 22: 0xeb50235e (stale) Mirror 23: 0xeb50235e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc6ad): Mirror 22: 0xeb50235e (stale) Mirror 23: 0xeb50235e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc6ad: skipped 000941 1788836308.735685 write 0x0249ed thru 0x025b26 (0x113a bytes) 000943 1788836308.740942 write 0x01d4f6 thru 0x01d5b4 (0xbf bytes) 000944 1788836308.757632 mapread 0x003da1 thru 0x00cdd6 (0x9036 bytes) 000945 1788836308.795339 mapread 0x00845e thru 0x00aa75 (0x2618 bytes) 000946 1788836308.797112 punch from 0x00940a to 0x00b41a (0x2011 bytes) 000947 1788836308.818978 MIRROR_RESYNC 3 mirrors 000949 1788836310.438615 punch from 0x013c5e to 0x01e9c6 (0xad69 bytes) 000951 1788836310.499985 mapwrite 0x014f5b thru 0x0175b8 (0x265e bytes) 000952 1788836310.583560 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x166:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25b27): Mirror 22: 0x195362e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b27): Mirror 22: 0x195362e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b27: skipped 000953 1788836310.909358 write 0x034db3 thru 0x03fc9f (0xaeed bytes) 000954 1788836310.910775 punch from 0x03f02d to 0x03fc9f (0xc73 bytes) 000956 1788836310.933111 read 0x021697 thru 0x02fbd0 (0xe53a bytes) 000958 1788836310.961391 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fca0): Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fca0): Mirror 22: 0x8579b763 (stale) CRC-32 checksum value for chunk [0, 0x3fca0): Mirror 1: 0x94371e1f Mirror 23: 0x94371e1f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fca0: skipped 000961 1788836312.415277 MIRROR_RESYNC 3 mirrors 000962 1788836313.681739 MIRROR_EXTEND to 4 mirrors 000964 1788836314.884859 MIRROR_EXTEND to 5 mirrors 000967 1788836315.712864 write 0x01b8b8 thru 0x01ff7d (0x46c6 bytes) 000968 1788836315.801395 mapwrite 0x000607 thru 0x00ba3e (0xb438 bytes) 000970 1788836315.885193 MIRROR_RESYNC 5 mirrors 000971 1788836318.680365 mapwrite 0x03f6b6 thru 0x03ffff (0x94a bytes) 000973 1788836318.865028 write 0x0199b5 thru 0x028ebf (0xf50b bytes) 000974 1788836318.877814 read 0x00c4a6 thru 0x0187ec (0xc347 bytes) 000975 1788836318.898564 mapwrite 0x0276ec thru 0x03750c (0xfe21 bytes) 000978 1788836319.050034 punch from 0x017880 to 0x025deb (0xe56c bytes) 000979 1788836319.058893 mapread 0x0211c7 thru 0x0248e2 (0x371c bytes) 000980 1788836319.082960 mapwrite 0x005d8b thru 0x00707c (0x12f2 bytes) 000982 1788836319.138199 mapread 0x028eb9 thru 0x02f96a (0x6ab2 bytes) 000983 1788836319.139700 mapwrite 0x03aae0 thru 0x03ffff (0x5520 bytes) 000984 1788836319.274930 write 0x00d7ff thru 0x016c8c (0x948e bytes) 000985 1788836319.276324 write 0x002ab9 thru 0x009feb (0x7533 bytes) 000987 1788836319.281602 read 0x03dc66 thru 0x03ffff (0x239a bytes) 000988 1788836319.282970 trunc from 0x040000 to 0x007d53 (0x382ac bytes) 000990 1788836319.326658 punch from 0x0008b1 to 0x005016 (0x4766 bytes) 000991 1788836319.367785 mapread 0x00582e thru 0x007d53 (0x2526 bytes) 000992 1788836319.391483 punch from 0x002742 to 0x003977 (0x1236 bytes) 000993 1788836319.397612 write 0x001da5 thru 0x00a817 (0x8a73 bytes) 000994 1788836319.400061 trunc from 0x00a818 to 0x00b769 (0xf52 bytes) 000995 1788836319.425888 mapwrite 0x016cf6 thru 0x021cc4 (0xafcf bytes) 000998 1788836319.523880 punch from 0x018feb to 0x0197ff (0x815 bytes) 000999 1788836319.531313 mapwrite 0x0362c5 thru 0x03a8b7 (0x45f3 bytes) 001000 1788836319.629932 mapwrite 0x00a99f thru 0x016914 (0xbf76 bytes) All operations completed A-OK! PASS 70a (197s) == sanity-flr test 71: check mirror extend parent fid ==== 22:58:45 (1788836325) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 22:58:49 (1788836329) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.361584 s, 5.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340475 s, 3.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x298:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x298:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x297:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 22:59:00 (1788836340) Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre2 Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 533408 .. Extending file size to 1513664 .. done Extending file size to 7825056 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6442368 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9285824 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7243136 .. Extending file size to 7684960 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8525824 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 57311 57312 57315 57316 57317 192.168.204.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg453-client.virtnet /mnt/lustre2 (opts:) 192.168.204.153@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg453-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:00:24 (1788836424) Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre2 Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6977024 .. Extending file size to 8020704 .. Extending file size to 8191424 .. Extending file size to 8494976 .. Extending file size to 9062976 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8769408 .. Extending file size to 9327328 .. Waiting 58748 58750 58751 58752 58756 done 192.168.204.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg453-client.virtnet /mnt/lustre2 (opts:) 192.168.204.153@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg453-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:01:46 (1788836506) Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127808 s, 16.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.337187 s, 6.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x285c5daa (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg453-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:02:07 (1788836527) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124813 s, 16.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 23:02:15 (1788836535) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ad:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2af:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:02:23 (1788836543) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.350863 s, 6.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:02:33 (1788836553) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269149 s, 7.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x295:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x296:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:02:42 (1788836562) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.326394 s, 6.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x297:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x297:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x298:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:03:17 (1788836597) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg453-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1193 s, 8.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x299:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x299:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:04:22 (1788836662) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg453-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.39869 s, 5.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:05:24 (1788836724) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg453-server mds-ost sync done. Creating new pool test_204f oleg453-server: Pool lustre.test_204f created Adding targets to pool oleg453-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg453-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21904 s, 9.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg453-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg453-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg453-server: Pool lustre.test_204f destroyed PASS 204f (84s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:06:48 (1788836808) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:06:56 (1788836816) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:07:04 (1788836824) oleg453-server: Pool lustre.test_206 created Adding targets to pool oleg453-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg453-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg453-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg453-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg453-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:07:30 (1788836850) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114111 s, 9.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465966 s, 22.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3c10dcb0 Mirror 3: 0x3c10dcb0 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:07:45 (1788836865) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (7s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:07:53 (1788836873) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (8s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:08:01 (1788836881) Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.212176 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1530000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x20c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x20c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x20c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x8f0000 Read this (4 bytes): BBBB 192.168.204.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg453-client.virtnet /mnt/lustre2 (opts:) PASS 209a (46s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:08:47 (1788836927) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0843327 s, 97.1 kB/s the very first read read with warm cache PASS 209b (12s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:09:00 (1788836940) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462981 s, 22.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d2 lmm_fid: [0x200000404:0x1d2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d2 lmm_fid: [0x200000404:0x1d2:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2a7:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834386 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 23:09:03.000000000 -0400 Modify: 2026-09-07 23:09:03.000000000 -0400 Change: 2026-09-07 23:09:05.000000000 -0400 Birth: 2026-09-07 23:09:04.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:09:12 (1788836952) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (5s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:09:16 (1788836956) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.767544 s, 53.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0790558 s, 51.8 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 7 23:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 7 23:09 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:09:27 (1788836967) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:09:37 (1788836977) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1900 sec =========== 23:09:48 (1788836988) === sanity-flr: start cleanup 23:09:50 (1788836990) === === sanity-flr: finish cleanup 23:09:57 (1788836997) ===