-----============= acceptance-small: sanity-flr ============----- Tue Sep 1 09:29:11 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg210-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 09:29:21 (1788269361) === oleg210-client.virtnet: executing check_config_client /mnt/lustre oleg210-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg210-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d8707e93000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d8707e93000.idle_timeout=debug disable quota as required oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 09:29:41 (1788269381) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7623] == sanity-flr test 0a: lfs mirror create with -N option == 09:29:42 (1788269382) PASS 0a (10s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 09:29:52 (1788269392) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 09:29:53 (1788269393) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 09:29:55 (1788269395) PASS 0d (14s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 09:30:09 (1788269409) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 09:30:11 (1788269411) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 09:30:12 (1788269412) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d8707e93000.stats=clear osc.lustre-OST0001-osc-ffff8d8707e93000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 09:30:16 (1788269416) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (5s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 09:30:21 (1788269421) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 09:30:25 (1788269425) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 09:30:27 (1788269427) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 09:30:32 (1788269432) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (5s) == sanity-flr test 2: create components from existing files ========================================================== 09:30:37 (1788269437) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 09:30:41 (1788269441) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (1s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 09:30:43 (1788269443) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 09:30:47 (1788269447) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 09:30:53 (1788269453) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 09:30:56 (1788269456) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.520381 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512988 s, 20.4 MB/s PASS 21 (6s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 09:31:02 (1788269462) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195 s, 53.8 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 09:31:06 (1788269466) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 09:31:12 (1788269472) Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8707e93000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8707e93000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d8707e93000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8707e93000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (35s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 09:31:46 (1788269506) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0944083 s, 33.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8707e93000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8707e93000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (20s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 09:32:06 (1788269526) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8707e93000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8707e93000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d8707e93000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8707e93000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (37s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 09:32:43 (1788269563) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.75723 s, 19.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 1 09:32 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.484s user 0m0.001s sys 0m0.169s Stopping /mnt/lustre-ost1 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d8707e93000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d8707e93000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.538s user 0m0.000s sys 0m0.174s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg210-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg210-server oleg210-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d8707e93000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8707e93000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.490s user 0m0.000s sys 0m0.181s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg210-server: oleg210-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg210-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg210-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d8707e93000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d8707e93000.ost_server_uuid in FULL state after 0 sec PASS 33b (45s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 09:33:28 (1788269608) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 09:33:30 (1788269610) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 09:33:32 (1788269612) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 09:33:33 (1788269613) PASS 35 (3s) == sanity-flr test 36a: write to mirrored files ========== 09:33:37 (1788269617) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.214419 s, 58.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (6s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 09:33:42 (1788269622) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149845 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143232 s, 73.2 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 09:33:45 (1788269625) PASS 36c (4s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 09:33:49 (1788269629) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000533166 s, 1.9 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000599446 s, 1.7 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000609423 s, 1.7 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000582234 s, 1.8 MB/s flv=10 olv=10 PASS 36d (3s) == sanity-flr test 37: mirror I/O API verification ======= 09:33:52 (1788269632) create test files with size 15737776 .. 59f7fd2db13c2793a3698d77520755a1 - c9fc9b50903577744f736cc49e18b46f - 8fe64664be8bdf5a6c1254c7c6f827c5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre PASS 37 (8s) == sanity-flr test 38: resync ============================ 09:34:00 (1788269640) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (7s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 09:34:07 (1788269647) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 09:34:10 (1788269650) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510312 s, 20.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148166 s, 70.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 09:34:14 (1788269654) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.048663 s, 43.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.141675 s, 29.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102551 s, 20.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (6s) == sanity-flr test 42: lfs mirror verify ================= 09:34:21 (1788269661) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 09:34:22 (1788269662) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 09:34:23 (1788269663) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164386 s, 63.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (4s) == sanity-flr test 44a: lfs mirror split check =========== 09:34:27 (1788269667) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (1s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 09:34:29 (1788269669) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (4s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 09:34:33 (1788269673) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.131381 s, 79.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 09:34:48 (1788269688) llite.lustre-ffff8d87036cd800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.141846 s, 73.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 Stopping client oleg210-client.virtnet /mnt/lustre (opts:) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 6 flags: 1 llite.lustre-ffff8d8705757000.xattr_cache=1 PASS 44e (7s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 09:34:55 (1788269695) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (18s) == sanity-flr test 46: Verify setstripe --copy option ==== 09:35:13 (1788269713) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (15s) == sanity-flr test 47: Verify mirror obj alloc =========== 09:35:28 (1788269728) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 09:35:30 (1788269730) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0925353 s, 34.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0768341 s, 40.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc31aa215 (nosync,stale) Mirror 2: 0xc31aa215 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 27ecc2436d2238de3238028cfc870b6f - mirror 1 checksum: 30d765137dfe762f283e00e3c8678949 - mirror 2 checksum: 27ecc2436d2238de3238028cfc870b6f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-01T09:35:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-01T09:35:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 27ecc2436d2238de3238028cfc870b6f - mirror 1 checksum: 30d765137dfe762f283e00e3c8678949 - mirror 2 checksum: 30d765137dfe762f283e00e3c8678949 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-01T09:35:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-01T09:35:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (4s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 09:35:35 (1788269735) Failing mds1 on oleg210-server Stopping /mnt/lustre-mds1 (opts:) on oleg210-server 09:35:38 (1788269738) shut down facet: mds1 facet_host: oleg210-server facet_failover_host: oleg210-server Failover mds1 to oleg210-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 09:35:53 (1788269753) targets are mounted 09:35:53 (1788269753) facet_failover done oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021699 s, 48.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (27s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 09:36:02 (1788269762) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (1s) == sanity-flr test 50b: mirror rsync handles sparseness == 09:36:03 (1788269763) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (1s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 09:36:04 (1788269764) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00421527 s, 3.9 MB/s PASS 50c (4s) == sanity-flr test 50d: mirror rsync keep holes ========== 09:36:08 (1788269768) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (1s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 09:36:09 (1788269769) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0038359 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0390145 s, 105 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 09:36:13 (1788269773) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (9s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 09:36:22 (1788269782) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (21s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 09:36:43 (1788269803) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 09:37:02 (1788269822) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 09:37:20 (1788269840) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (4s) == sanity-flr test 70: mirror create and split race ====== 09:37:24 (1788269844) mirror create pid 54225 mirror split pid 54226 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54225 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54226 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 09:38:26 (1788269906) Seed set to 9907 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003 1788269907.652920 MIRROR_EXTEND to 2 mirrors 000004 1788269907.835970 mapwrite 0x01f230 thru 0x020b3f (0x1910 bytes) 000005 1788269907.941792 read 0x0026c6 thru 0x00b5d8 (0x8f13 bytes) 000007 1788269907.948280 write 0x02a100 thru 0x02da51 (0x3952 bytes) 000010 1788269907.950220 mapwrite 0x02c3b6 thru 0x035734 (0x937f bytes) 000017 1788269908.013832 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35735): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35735): Mirror 2: 0xed11338 (stale) CRC-32 checksum value for chunk [0, 0x35735): Mirror 1: 0x75c4add7 Mirror 3: 0x75c4add7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35735: skipped 000018 1788269908.472083 write 0x03deb1 thru 0x03ffff (0x214f bytes) 000020 1788269908.502470 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xb640a2d0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96760cac Mirror 4: 0x96760cac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x165d2 000022 1788269909.065792 trunc from 0x040000 to 0x0165d1 (0x29a2e bytes) truncating to largest ever: 0x2d236 000024 1788269909.127514 trunc from 0x0165d2 to 0x02d235 (0x16c64 bytes) 000025 1788269909.141340 mapread 0x007c15 thru 0x008825 (0xc11 bytes) 000026 1788269909.146571 mapread 0x0232dc thru 0x0238cb (0x5f0 bytes) 000027 1788269909.151539 write 0x02bcda thru 0x02fe68 (0x418f bytes) 000028 1788269909.155289 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fe69): Mirror 2: 0x1cc5b655 (stale) Mirror 3: 0x87abe2e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fe69): Mirror 2: 0x1cc5b655 (stale) Mirror 3: 0x87abe2e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fe69: skipped 000030 1788269909.319246 read 0x00b2ec thru 0x00b463 (0x178 bytes) 000031 1788269909.321901 mapread 0x00872b thru 0x011249 (0x8b1f bytes) 000032 1788269909.333591 write 0x018126 thru 0x0245d2 (0xc4ad bytes) 000034 1788269909.340277 trunc from 0x02fe69 to 0x02b932 (0x4536 bytes) 000040 1788269909.349699 mapread 0x00de05 thru 0x01d7db (0xf9d7 bytes) 000041 1788269909.356510 read 0x026834 thru 0x027bb5 (0x1382 bytes) 000043 1788269909.363219 mapread 0x01aade thru 0x024864 (0x9d87 bytes) 000045 1788269909.364273 mapread 0x01432b thru 0x01753d (0x3213 bytes) 000046 1788269909.365038 mapread 0x00c845 thru 0x0177f2 (0xafae bytes) 000047 1788269909.365999 mapread 0x008e7c thru 0x014b95 (0xbd1a bytes) 000048 1788269909.366872 mapread 0x01f9c8 thru 0x02b932 (0xbf6b bytes) 000050 1788269909.368387 mapwrite 0x03f346 thru 0x03ffff (0xcba bytes) 000053 1788269909.414842 MIRROR_RESYNC 3 mirrors 000054 1788269910.064037 write 0x02c181 thru 0x03bfaf (0xfe2f bytes) 000057 1788269910.101975 read 0x00069c thru 0x004339 (0x3c9e bytes) 000058 1788269910.108692 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b0456f7 (stale) Mirror 3: 0x4b0456f7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd4cbda75 Mirror 4: 0xd4cbda75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000059 1788269910.643220 write 0x0184a3 thru 0x0210e1 (0x8c3f bytes) 000060 1788269910.701133 write 0x01cb21 thru 0x02c61f (0xfaff bytes) 000062 1788269910.709058 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b0456f7 (stale) Mirror 3: 0x4b0456f7 (stale) Mirror 4: 0xd4cbda75 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x95dfaec Mirror 5: 0x95dfaec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000064 1788269911.285998 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4b0456f7 (stale) Mirror 3: 0x4b0456f7 (stale) Mirror 4: 0xd4cbda75 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x95dfaec Mirror 5: 0x95dfaec Mirror 6: 0x95dfaec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000065 1788269911.725233 trunc from 0x040000 to 0x010236 (0x2fdc9 bytes) 000066 1788269911.752783 read 0x002f07 thru 0x00b3bc (0x84b6 bytes) 000068 1788269911.757615 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x10237): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10237): Mirror 2: 0xabf721b9 (stale) Mirror 3: 0xabf721b9 (stale) Mirror 4: 0xabf721b9 (stale) Mirror 5: 0xabf721b9 (stale) Mirror 6: 0xabf721b9 (stale) CRC-32 checksum value for chunk [0, 0x10237): Mirror 1: 0xabf721b9 Mirror 7: 0xabf721b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10237: skipped 000071 1788269912.221316 MIRROR_RESYNC 7 mirrors 000073 1788269912.741058 write 0x036136 thru 0x039a17 (0x38e2 bytes) 000075 1788269912.777545 write 0x019384 thru 0x027b39 (0xe7b6 bytes) 000076 1788269912.791128 MIRROR_RESYNC 7 mirrors 000077 1788269913.926166 mapwrite 0x03cbc3 thru 0x03ffff (0x343d bytes) 000079 1788269914.038766 mapread 0x032eee thru 0x03ffff (0xd112 bytes) 000084 1788269914.049873 mapwrite 0x011d63 thru 0x01692a (0x4bc8 bytes) 000085 1788269914.093807 write 0x022020 thru 0x031420 (0xf401 bytes) 000086 1788269914.104887 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb387c5a8 (stale) Mirror 3: 0xb387c5a8 (stale) Mirror 4: 0xb387c5a8 (stale) Mirror 5: 0xb387c5a8 (stale) Mirror 6: 0xb387c5a8 (stale) Mirror 7: 0xb387c5a8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x49bcd03e Mirror 8: 0x49bcd03e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000089 1788269914.808865 mapwrite 0x003ac4 thru 0x00c0fd (0x863a bytes) 000091 1788269914.911448 write 0x026112 thru 0x02841a (0x2309 bytes) 000092 1788269914.917874 trunc from 0x040000 to 0x016ee1 (0x2911e bytes) 000093 1788269914.934739 read 0x010fda thru 0x016ee1 (0x5f08 bytes) 000094 1788269914.948214 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x16ee2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16ee2): Mirror 2: 0x33da03af (stale) Mirror 3: 0x33da03af (stale) Mirror 4: 0x33da03af (stale) Mirror 5: 0x33da03af (stale) Mirror 6: 0x33da03af (stale) Mirror 7: 0x33da03af (stale) Mirror 8: 0x7b87c052 (stale) CRC-32 checksum value for chunk [0, 0x16ee2): Mirror 1: 0x133b08a Mirror 9: 0x133b08a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16ee2: skipped 000096 1788269915.488833 write 0x0292d0 thru 0x037fe3 (0xed14 bytes) 000099 1788269915.524448 read 0x02fa44 thru 0x02fba9 (0x166 bytes) 000101 1788269915.528280 MIRROR_RESYNC 9 mirrors 000102 1788269917.131103 write 0x023367 thru 0x02e326 (0xafc0 bytes) 000103 1788269917.174137 write 0x020fde thru 0x023193 (0x21b6 bytes) 000105 1788269917.180053 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x37fe4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fe4): Mirror 2: 0x45234b72 (stale) Mirror 3: 0x45234b72 (stale) Mirror 4: 0x45234b72 (stale) Mirror 5: 0x45234b72 (stale) Mirror 6: 0x45234b72 (stale) Mirror 7: 0x45234b72 (stale) Mirror 8: 0x45234b72 (stale) Mirror 9: 0x45234b72 (stale) CRC-32 checksum value for chunk [0, 0x37fe4): Mirror 1: 0xb7a645b7 Mirror 10: 0xb7a645b7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fe4: skipped truncating to largest ever: 0x314a4 000107 1788269917.774573 trunc from 0x037fe4 to 0x0314a3 (0x6b40 bytes) 000109 1788269917.812224 mapread 0x00d85c thru 0x00e62b (0xdd0 bytes) 000112 1788269917.817214 write 0x026707 thru 0x0291b1 (0x2aab bytes) 000113 1788269917.820955 mapread 0x00eac5 thru 0x01323a (0x4776 bytes) 000117 1788269917.826355 mapwrite 0x03a14b thru 0x03fd35 (0x5beb bytes) 000118 1788269917.866475 write 0x016882 thru 0x020401 (0x9b80 bytes) truncating to largest ever: 0x3780b 000120 1788269917.870930 trunc from 0x03fd36 to 0x03780a (0x852b bytes) 000121 1788269917.885709 trunc from 0x03780b to 0x025163 (0x126a7 bytes) 000122 1788269917.895208 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x25164): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25164): Mirror 2: 0x4dd6ae63 (stale) Mirror 3: 0x4dd6ae63 (stale) Mirror 4: 0x4dd6ae63 (stale) Mirror 5: 0x4dd6ae63 (stale) Mirror 6: 0x4dd6ae63 (stale) Mirror 7: 0x4dd6ae63 (stale) Mirror 8: 0x4dd6ae63 (stale) Mirror 9: 0x4dd6ae63 (stale) Mirror 10: 0x1d11d7de (stale) CRC-32 checksum value for chunk [0, 0x25164): Mirror 1: 0xdda37692 Mirror 11: 0xdda37692 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25164: skipped 000124 1788269918.533976 trunc from 0x025164 to 0x028ca4 (0x3b41 bytes) 000126 1788269918.603411 write 0x013745 thru 0x01aabb (0x7377 bytes) 000128 1788269918.610418 mapread 0x021767 thru 0x026929 (0x51c3 bytes) 000129 1788269918.619481 read 0x0240e5 thru 0x028ca4 (0x4bc0 bytes) 000130 1788269918.622772 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x28ca5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ca5): Mirror 2: 0x17d0aa1b (stale) Mirror 3: 0x17d0aa1b (stale) Mirror 4: 0x17d0aa1b (stale) Mirror 5: 0x17d0aa1b (stale) Mirror 6: 0x17d0aa1b (stale) Mirror 7: 0x17d0aa1b (stale) Mirror 8: 0x17d0aa1b (stale) Mirror 9: 0x17d0aa1b (stale) Mirror 10: 0x42bf784 (stale) Mirror 11: 0xbaa56a89 (stale) CRC-32 checksum value for chunk [0, 0x28ca5): Mirror 1: 0xac6b6723 Mirror 12: 0xac6b6723 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ca5: skipped 000131 1788269919.329965 trunc from 0x028ca5 to 0x02350b (0x5799 bytes) 000132 1788269919.396560 read 0x006568 thru 0x0139af (0xd448 bytes) 000134 1788269919.406004 write 0x02956d thru 0x02ec9b (0x572f bytes) 000137 1788269919.408050 write 0x01997f thru 0x01b463 (0x1ae5 bytes) 000138 1788269919.415030 trunc from 0x02ec9c to 0x024afc (0xa19f bytes) 000140 1788269919.437517 write 0x035d7e thru 0x03fbfb (0x9e7e bytes) 000141 1788269919.438874 write 0x00ed3e thru 0x0108e9 (0x1bac bytes) 000145 1788269919.443143 write 0x00b316 thru 0x01421d (0x8f08 bytes) 000146 1788269919.450775 mapread 0x034d43 thru 0x0352b7 (0x575 bytes) 000147 1788269919.454035 mapread 0x001f23 thru 0x011ac5 (0xfba3 bytes) truncating to largest ever: 0x3c890 000150 1788269919.467779 trunc from 0x03fbfc to 0x03c88f (0x336c bytes) 000151 1788269919.482031 write 0x018136 thru 0x019c85 (0x1b50 bytes) 000153 1788269919.489966 mapwrite 0x00ba14 thru 0x00cfbc (0x15a9 bytes) 000154 1788269919.528218 write 0x0373b4 thru 0x037baf (0x7fc bytes) 000155 1788269919.529048 mapread 0x018282 thru 0x02592b (0xd6aa bytes) 000156 1788269919.548320 trunc from 0x03c890 to 0x039d5e (0x2b31 bytes) 000157 1788269919.568219 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39d5f): Mirror 2: 0xc2ecb520 (stale) Mirror 3: 0xc2ecb520 (stale) Mirror 4: 0xc2ecb520 (stale) Mirror 5: 0xc2ecb520 (stale) Mirror 6: 0xc2ecb520 (stale) Mirror 7: 0xc2ecb520 (stale) Mirror 8: 0xc2ecb520 (stale) Mirror 9: 0xc2ecb520 (stale) Mirror 10: 0x694f1856 (stale) Mirror 11: 0x24332c0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d5f): Mirror 2: 0xc2ecb520 (stale) Mirror 3: 0xc2ecb520 (stale) Mirror 4: 0xc2ecb520 (stale) Mirror 5: 0xc2ecb520 (stale) Mirror 6: 0xc2ecb520 (stale) Mirror 7: 0xc2ecb520 (stale) Mirror 8: 0xc2ecb520 (stale) Mirror 9: 0xc2ecb520 (stale) Mirror 10: 0x694f1856 (stale) Mirror 11: 0x24332c0d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d5f: skipped 000158 1788269919.930798 read 0x02a3f2 thru 0x02d0f3 (0x2d02 bytes) 000160 1788269919.935287 write 0x029ac2 thru 0x036b79 (0xd0b8 bytes) 000163 1788269919.942358 read 0x023da6 thru 0x02a80e (0x6a69 bytes) 000166 1788269919.948890 read 0x039282 thru 0x039d5e (0xadd bytes) 000167 1788269919.949496 mapread 0x0349b5 thru 0x0362d9 (0x1925 bytes) 000168 1788269919.950502 write 0x03a19a thru 0x03ffff (0x5e66 bytes) 000171 1788269919.952525 write 0x00806e thru 0x00f34b (0x72de bytes) 000173 1788269919.953959 mapwrite 0x03f9b4 thru 0x03ffff (0x64c bytes) 000174 1788269919.980629 read 0x02a3bc thru 0x030425 (0x606a bytes) 000175 1788269919.981265 read 0x01b2e2 thru 0x0284ab (0xd1ca bytes) 000176 1788269919.981754 read 0x00214f thru 0x006f10 (0x4dc2 bytes) 000177 1788269919.982126 read 0x01ecf9 thru 0x02086a (0x1b72 bytes) 000178 1788269919.986563 mapwrite 0x025e9b thru 0x027caa (0x1e10 bytes) 000179 1788269920.010983 read 0x03c03f thru 0x03ffff (0x3fc1 bytes) 000180 1788269920.011645 mapread 0x020c84 thru 0x02f8d1 (0xec4e bytes) 000182 1788269920.013035 mapread 0x008e48 thru 0x00ee70 (0x6029 bytes) 000185 1788269920.015160 write 0x022093 thru 0x026a7f (0x49ed bytes) 000186 1788269920.019749 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7b7b6113 (stale) Mirror 4: 0x7b7b6113 (stale) Mirror 5: 0x7b7b6113 (stale) Mirror 6: 0x7b7b6113 (stale) Mirror 7: 0x7b7b6113 (stale) Mirror 8: 0x7b7b6113 (stale) Mirror 9: 0x7b7b6113 (stale) Mirror 10: 0xbad52352 (stale) Mirror 11: 0xce0dfeef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7b7b6113 (stale) Mirror 4: 0x7b7b6113 (stale) Mirror 5: 0x7b7b6113 (stale) Mirror 6: 0x7b7b6113 (stale) Mirror 7: 0x7b7b6113 (stale) Mirror 8: 0x7b7b6113 (stale) Mirror 9: 0x7b7b6113 (stale) Mirror 10: 0xbad52352 (stale) Mirror 11: 0xce0dfeef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000187 1788269920.370408 mapwrite 0x00bd3b thru 0x019995 (0xdc5b bytes) 000188 1788269920.403983 read 0x03dfdb thru 0x03ffff (0x2025 bytes) 000189 1788269920.412364 trunc from 0x040000 to 0x03b639 (0x49c6 bytes) 000190 1788269920.428511 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b63a): Mirror 4: 0xccffb2b0 (stale) Mirror 5: 0xccffb2b0 (stale) Mirror 6: 0xccffb2b0 (stale) Mirror 7: 0xccffb2b0 (stale) Mirror 8: 0xccffb2b0 (stale) Mirror 9: 0xccffb2b0 (stale) Mirror 10: 0xbcb3f72f (stale) Mirror 11: 0xc37b0f26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b63a): Mirror 4: 0xccffb2b0 (stale) Mirror 5: 0xccffb2b0 (stale) Mirror 6: 0xccffb2b0 (stale) Mirror 7: 0xccffb2b0 (stale) Mirror 8: 0xccffb2b0 (stale) Mirror 9: 0xccffb2b0 (stale) Mirror 10: 0xbcb3f72f (stale) Mirror 11: 0xc37b0f26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b63a: skipped 000191 1788269920.705195 read 0x016e04 thru 0x01a72b (0x3928 bytes) 000192 1788269920.710585 trunc from 0x03b63a to 0x023481 (0x181b8 bytes) 000194 1788269920.716882 read 0x0075ec thru 0x00f4ef (0x7f04 bytes) 000195 1788269920.723654 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x23482): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23482): Mirror 4: 0xdc3a1a05 (stale) Mirror 5: 0xdc3a1a05 (stale) Mirror 6: 0xdc3a1a05 (stale) Mirror 7: 0xdc3a1a05 (stale) Mirror 8: 0xdc3a1a05 (stale) Mirror 9: 0xdc3a1a05 (stale) Mirror 10: 0xb1951474 (stale) Mirror 11: 0xd0c017c8 (stale) CRC-32 checksum value for chunk [0, 0x23482): Mirror 1: 0xe253168f Mirror 12: 0xe253168f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23482: skipped 000196 1788269921.215348 read 0x01bcec thru 0x023481 (0x7796 bytes) 000197 1788269921.221379 write 0x01280b thru 0x0151f3 (0x29e9 bytes) 000198 1788269921.254740 mapwrite 0x00ea13 thru 0x016d59 (0x8347 bytes) 000199 1788269921.296722 write 0x03c2be thru 0x03ffff (0x3d42 bytes) 000200 1788269921.299962 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7b7b6113 (stale) Mirror 6: 0x7b7b6113 (stale) Mirror 7: 0x7b7b6113 (stale) Mirror 8: 0x7b7b6113 (stale) Mirror 9: 0x7b7b6113 (stale) Mirror 10: 0xbad52352 (stale) Mirror 11: 0xce0dfeef (stale) Mirror 12: 0x8a6456a3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7b7b6113 (stale) Mirror 6: 0x7b7b6113 (stale) Mirror 7: 0x7b7b6113 (stale) Mirror 8: 0x7b7b6113 (stale) Mirror 9: 0x7b7b6113 (stale) Mirror 10: 0xbad52352 (stale) Mirror 11: 0xce0dfeef (stale) Mirror 12: 0x8a6456a3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000202 1788269921.567878 read 0x025e7c thru 0x03230f (0xc494 bytes) 000203 1788269921.572460 mapwrite 0x03bcca thru 0x03ffff (0x4336 bytes) 000205 1788269921.615907 mapwrite 0x018a1f thru 0x01cff1 (0x45d3 bytes) 000206 1788269921.650665 write 0x023236 thru 0x026d64 (0x3b2f bytes) 000207 1788269921.659034 mapread 0x02e444 thru 0x030287 (0x1e44 bytes) 000208 1788269921.659759 mapwrite 0x03db96 thru 0x03ffff (0x246a bytes) 000209 1788269921.687277 write 0x02d9b0 thru 0x034207 (0x6858 bytes) 000210 1788269921.694687 read 0x025c51 thru 0x02efce (0x937e bytes) 000213 1788269921.700260 MIRROR_RESYNC 9 mirrors 000214 1788269923.199022 read 0x03a48f thru 0x03ffff (0x5b71 bytes) 000219 1788269923.203100 write 0x00affd thru 0x018e07 (0xde0b bytes) 000220 1788269923.254511 trunc from 0x040000 to 0x0267a8 (0x19857 bytes) 000221 1788269923.264342 mapwrite 0x03c3a9 thru 0x03ffff (0x3c57 bytes) 000222 1788269923.307506 write 0x027f23 thru 0x02eef9 (0x6fd7 bytes) 000223 1788269923.313507 write 0x0094ec thru 0x00b101 (0x1c16 bytes) 000224 1788269923.316843 write 0x038988 thru 0x03f01d (0x6696 bytes) 000229 1788269923.319419 trunc from 0x040000 to 0x00329d (0x3cd62 bytes) 000231 1788269923.327588 trunc from 0x00329e to 0x023542 (0x202a5 bytes) 000232 1788269923.338968 mapread 0x0004f8 thru 0x00b89d (0xb3a6 bytes) 000233 1788269923.346803 MIRROR_RESYNC 9 mirrors 000234 1788269924.229660 write 0x03a14e thru 0x03ffff (0x5eb2 bytes) 000235 1788269924.258003 write 0x018754 thru 0x0196e0 (0xf8d bytes) 000238 1788269924.262943 mapread 0x02e8b0 thru 0x03816f (0x98c0 bytes) 000243 1788269924.278644 mapread 0x012ba2 thru 0x015fed (0x344c bytes) 000245 1788269924.288783 mapwrite 0x033426 thru 0x038aef (0x56ca bytes) 000247 1788269924.315732 write 0x0185c7 thru 0x018d41 (0x77b bytes) 000248 1788269924.316562 mapwrite 0x02c701 thru 0x038fc4 (0xc8c4 bytes) 000249 1788269924.343928 write 0x00517e thru 0x00c39a (0x721d bytes) 000253 1788269924.350551 trunc from 0x040000 to 0x034912 (0xb6ed bytes) 000255 1788269924.368977 read 0x01f171 thru 0x020313 (0x11a3 bytes) 000256 1788269924.372865 read 0x00cc07 thru 0x01298e (0x5d88 bytes) 000257 1788269924.373447 trunc from 0x034913 to 0x001845 (0x330cd bytes) 000258 1788269924.396020 MIRROR_RESYNC 9 mirrors 000259 1788269924.829463 mapread 0x001785 thru 0x001845 (0xc1 bytes) 000260 1788269924.832121 mapwrite 0x02cb0b thru 0x030b18 (0x400e bytes) 000262 1788269924.890057 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x30b19): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30b19): Mirror 5: 0x224de3df (stale) Mirror 6: 0x224de3df (stale) Mirror 7: 0x224de3df (stale) Mirror 8: 0x224de3df (stale) Mirror 9: 0x224de3df (stale) Mirror 10: 0x224de3df (stale) Mirror 11: 0x224de3df (stale) Mirror 12: 0x224de3df (stale) CRC-32 checksum value for chunk [0, 0x30b19): Mirror 1: 0x58fd4cdb Mirror 13: 0x58fd4cdb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30b19: skipped 000263 1788269925.287923 mapread 0x01b9fb thru 0x01fe8a (0x4490 bytes) 000265 1788269925.318498 trunc from 0x030b19 to 0x019034 (0x17ae4 bytes) 000267 1788269925.358231 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19035): Mirror 5: 0xfd3fda19 (stale) Mirror 6: 0xfd3fda19 (stale) Mirror 8: 0xfd3fda19 (stale) Mirror 9: 0xfd3fda19 (stale) Mirror 10: 0xfd3fda19 (stale) Mirror 11: 0xfd3fda19 (stale) Mirror 12: 0xfd3fda19 (stale) Mirror 13: 0xfd3fda19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19035): Mirror 5: 0xfd3fda19 (stale) Mirror 6: 0xfd3fda19 (stale) Mirror 8: 0xfd3fda19 (stale) Mirror 9: 0xfd3fda19 (stale) Mirror 10: 0xfd3fda19 (stale) Mirror 11: 0xfd3fda19 (stale) Mirror 12: 0xfd3fda19 (stale) Mirror 13: 0xfd3fda19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19035: skipped 000268 1788269925.584327 mapread 0x0002b3 thru 0x00eda5 (0xeaf3 bytes) 000269 1788269925.592463 read 0x0028d2 thru 0x008c96 (0x63c5 bytes) 000271 1788269925.593540 mapwrite 0x016200 thru 0x01cbf9 (0x69fa bytes) 000272 1788269925.625800 mapread 0x016dbe thru 0x01cbf9 (0x5e3c bytes) 000273 1788269925.626554 write 0x03194e thru 0x0383e9 (0x6a9c bytes) 000274 1788269925.627719 write 0x01a290 thru 0x01b460 (0x11d1 bytes) 000277 1788269925.629448 trunc from 0x0383ea to 0x02231f (0x160ca bytes) 000278 1788269925.635922 read 0x005aa1 thru 0x0149b9 (0xef19 bytes) 000279 1788269925.636427 mapwrite 0x01a677 thru 0x01cdd8 (0x2762 bytes) 000280 1788269925.653085 trunc from 0x022320 to 0x01626a (0xc0b5 bytes) 000281 1788269925.661571 write 0x004a40 thru 0x008c17 (0x41d8 bytes) 000284 1788269925.663343 trunc from 0x01626b to 0x010a0b (0x585f bytes) 000285 1788269925.672071 mapread 0x00c4f4 thru 0x010a0b (0x4518 bytes) 000286 1788269925.672797 trunc from 0x010a0c to 0x03980c (0x28e01 bytes) 000289 1788269925.679289 trunc from 0x03980d to 0x01ce7b (0x1c991 bytes) 000291 1788269925.685015 trunc from 0x01ce7c to 0x02f80e (0x12993 bytes) 000293 1788269925.693380 MIRROR_RESYNC 9 mirrors 000294 1788269926.827217 write 0x01a274 thru 0x01d080 (0x2e0d bytes) 000296 1788269926.880474 mapread 0x0064a3 thru 0x00c0e6 (0x5c44 bytes) 000297 1788269926.900427 mapwrite 0x02cc1e thru 0x033ed1 (0x72b4 bytes) 000298 1788269926.963059 mapread 0x0170f5 thru 0x0210d3 (0x9fdf bytes) 000299 1788269926.977155 mapwrite 0x036afd thru 0x03f47b (0x897f bytes) truncating to largest ever: 0x3da1d 000300 1788269927.032927 trunc from 0x03f47c to 0x03da1c (0x1a5f bytes) 000303 1788269927.046871 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3da1d): Mirror 6: 0x46ac7f6a (stale) Mirror 8: 0x46ac7f6a (stale) Mirror 9: 0x46ac7f6a (stale) Mirror 10: 0x46ac7f6a (stale) Mirror 11: 0x46ac7f6a (stale) Mirror 12: 0x46ac7f6a (stale) Mirror 13: 0x46ac7f6a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da1d): Mirror 6: 0x46ac7f6a (stale) Mirror 8: 0x46ac7f6a (stale) Mirror 9: 0x46ac7f6a (stale) Mirror 10: 0x46ac7f6a (stale) Mirror 11: 0x46ac7f6a (stale) Mirror 12: 0x46ac7f6a (stale) Mirror 13: 0x46ac7f6a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da1d: skipped 000304 1788269927.340323 mapwrite 0x029ed3 thru 0x0364ee (0xc61c bytes) 000305 1788269927.414781 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3da1d): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da1d): Mirror 6: 0x46ac7f6a (stale) Mirror 8: 0x46ac7f6a (stale) Mirror 9: 0x46ac7f6a (stale) Mirror 10: 0x46ac7f6a (stale) Mirror 11: 0x46ac7f6a (stale) Mirror 12: 0x46ac7f6a (stale) Mirror 13: 0x46ac7f6a (stale) CRC-32 checksum value for chunk [0, 0x3da1d): Mirror 1: 0x146ed101 Mirror 14: 0x146ed101 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da1d: skipped 000306 1788269928.066000 trunc from 0x03da1d to 0x0103bd (0x2d65f bytes) 000307 1788269928.149264 read 0x00679d thru 0x009a85 (0x32e9 bytes) 000308 1788269928.153551 mapread 0x0060e9 thru 0x0103bd (0xa2d5 bytes) 000309 1788269928.172040 mapread 0x00b7d9 thru 0x0103bd (0x4be5 bytes) 000310 1788269928.175497 mapread 0x00ec6f thru 0x0103bd (0x174f bytes) 000311 1788269928.176114 mapwrite 0x036976 thru 0x03926b (0x28f6 bytes) 000314 1788269928.250660 write 0x03b31f thru 0x03ffff (0x4ce1 bytes) 000316 1788269928.251898 read 0x016068 thru 0x021596 (0xb52f bytes) 000317 1788269928.260520 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xef75b9c0 (stale) Mirror 8: 0xef75b9c0 (stale) Mirror 9: 0xef75b9c0 (stale) Mirror 10: 0xef75b9c0 (stale) Mirror 11: 0xef75b9c0 (stale) Mirror 12: 0xef75b9c0 (stale) Mirror 13: 0xef75b9c0 (stale) Mirror 14: 0x3cc9e18a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdd13af48 Mirror 15: 0xdd13af48 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000318 1788269928.862626 read 0x00d523 thru 0x019bb8 (0xc696 bytes) 000320 1788269928.873761 write 0x019815 thru 0x019c56 (0x442 bytes) 000322 1788269928.920651 mapread 0x01df19 thru 0x02a9b6 (0xca9e bytes) 000323 1788269928.941493 write 0x035017 thru 0x03ffff (0xafe9 bytes) 000328 1788269928.947495 read 0x004bd8 thru 0x0078cb (0x2cf4 bytes) 000329 1788269928.954643 mapwrite 0x029a3e thru 0x035900 (0xbec3 bytes) 000334 1788269929.000738 trunc from 0x040000 to 0x01a6e8 (0x25917 bytes) 000335 1788269929.042937 read 0x00c594 thru 0x0196fb (0xd168 bytes) 000336 1788269929.043998 mapread 0x014fd6 thru 0x017086 (0x20b1 bytes) 000337 1788269929.046957 write 0x02af44 thru 0x035162 (0xa21f bytes) 000339 1788269929.048636 write 0x03e76b thru 0x03ffff (0x1895 bytes) 000341 1788269929.050217 read 0x03f654 thru 0x03ffff (0x9ac bytes) 000342 1788269929.052344 trunc from 0x040000 to 0x03d439 (0x2bc6 bytes) 000344 1788269929.065021 mapwrite 0x028cc2 thru 0x02913e (0x47d bytes) 000345 1788269929.122612 trunc from 0x03d43a to 0x0049f1 (0x38a48 bytes) 000346 1788269929.142132 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x49f2): Mirror 8: 0x4cbe2e2b (stale) Mirror 9: 0x4cbe2e2b (stale) Mirror 10: 0x4cbe2e2b (stale) Mirror 11: 0x4cbe2e2b (stale) Mirror 12: 0x4cbe2e2b (stale) Mirror 13: 0x4cbe2e2b (stale) Mirror 14: 0x4cbe2e2b (stale) Mirror 15: 0x4cbe2e2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x49f2): Mirror 8: 0x4cbe2e2b (stale) Mirror 9: 0x4cbe2e2b (stale) Mirror 10: 0x4cbe2e2b (stale) Mirror 11: 0x4cbe2e2b (stale) Mirror 12: 0x4cbe2e2b (stale) Mirror 13: 0x4cbe2e2b (stale) Mirror 14: 0x4cbe2e2b (stale) Mirror 15: 0x4cbe2e2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x49f2: skipped 000349 1788269929.460510 mapread 0x00324e thru 0x0049f1 (0x17a4 bytes) 000350 1788269929.467780 write 0x021222 thru 0x026cb1 (0x5a90 bytes) 000351 1788269929.469067 mapwrite 0x015304 thru 0x01e2a9 (0x8fa6 bytes) 000353 1788269929.512982 trunc from 0x026cb2 to 0x0305c7 (0x9916 bytes) 000355 1788269929.535588 trunc from 0x0305c8 to 0x01a4a0 (0x16127 bytes) 000357 1788269929.552987 read 0x005dc9 thru 0x0120aa (0xc2e2 bytes) 000360 1788269929.559739 trunc from 0x01a4a1 to 0x01067c (0x9e24 bytes) 000361 1788269929.572458 mapwrite 0x03d8b1 thru 0x03ffff (0x274f bytes) 000362 1788269929.654991 trunc from 0x040000 to 0x005d4c (0x3a2b3 bytes) 000363 1788269929.668829 mapread 0x0012b1 thru 0x005d4c (0x4a9c bytes) 000367 1788269929.679303 mapwrite 0x01c843 thru 0x01ebbe (0x237c bytes) 000370 1788269929.735107 mapwrite 0x0130e0 thru 0x019900 (0x6821 bytes) 000372 1788269929.782792 mapwrite 0x00cef0 thru 0x010148 (0x3259 bytes) 000374 1788269929.820607 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ebbf): Mirror 8: 0xaff36b95 (stale) Mirror 9: 0xaff36b95 (stale) Mirror 10: 0xaff36b95 (stale) Mirror 11: 0xaff36b95 (stale) Mirror 12: 0xaff36b95 (stale) Mirror 13: 0xaff36b95 (stale) Mirror 15: 0xaff36b95 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ebbf): Mirror 8: 0xaff36b95 (stale) Mirror 9: 0xaff36b95 (stale) Mirror 10: 0xaff36b95 (stale) Mirror 11: 0xaff36b95 (stale) Mirror 12: 0xaff36b95 (stale) Mirror 13: 0xaff36b95 (stale) Mirror 15: 0xaff36b95 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ebbf: skipped 000377 1788269930.092972 MIRROR_RESYNC 8 mirrors 000378 1788269931.677647 read 0x01713a thru 0x01bd10 (0x4bd7 bytes) 000380 1788269931.685752 read 0x00bbc6 thru 0x00fd26 (0x4161 bytes) 000381 1788269931.696966 mapwrite 0x00e0f3 thru 0x0112a7 (0x31b5 bytes) 000382 1788269931.791808 mapread 0x009c3a thru 0x0158b6 (0xbc7d bytes) 000385 1788269931.839860 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ebbf): Mirror 9: 0x6b5770aa (stale) Mirror 10: 0x6b5770aa (stale) Mirror 11: 0x6b5770aa (stale) Mirror 12: 0x6b5770aa (stale) Mirror 13: 0x6b5770aa (stale) Mirror 15: 0x6b5770aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ebbf): Mirror 9: 0x6b5770aa (stale) Mirror 10: 0x6b5770aa (stale) Mirror 11: 0x6b5770aa (stale) Mirror 12: 0x6b5770aa (stale) Mirror 13: 0x6b5770aa (stale) Mirror 15: 0x6b5770aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ebbf: skipped 000387 1788269932.075975 write 0x0074fe thru 0x00ae3d (0x3940 bytes) 000388 1788269932.083758 write 0x01a0ad thru 0x028ff7 (0xef4b bytes) 000390 1788269932.087926 write 0x036f5b thru 0x03ffff (0x90a5 bytes) 000391 1788269932.089117 mapread 0x00eac7 thru 0x016b04 (0x803e bytes) 000392 1788269932.113657 mapread 0x01d962 thru 0x021468 (0x3b07 bytes) 000393 1788269932.114495 read 0x039aea thru 0x03ce6c (0x3383 bytes) 000399 1788269932.117045 mapwrite 0x021daf thru 0x027124 (0x5376 bytes) 000401 1788269932.155054 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1b6a19a6 (stale) Mirror 10: 0x1b6a19a6 (stale) Mirror 11: 0x1b6a19a6 (stale) Mirror 12: 0x1b6a19a6 (stale) Mirror 15: 0x1b6a19a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1b6a19a6 (stale) Mirror 10: 0x1b6a19a6 (stale) Mirror 11: 0x1b6a19a6 (stale) Mirror 12: 0x1b6a19a6 (stale) Mirror 15: 0x1b6a19a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000402 1788269932.330928 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x1b6a19a6 (stale) Mirror 10: 0x1b6a19a6 (stale) Mirror 11: 0x1b6a19a6 (stale) Mirror 12: 0x1b6a19a6 (stale) Mirror 15: 0x1b6a19a6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9309476 Mirror 16: 0x9309476 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000403 1788269932.814237 mapread 0x003ade thru 0x004747 (0xc6a bytes) 000405 1788269932.824830 write 0x008f78 thru 0x018a18 (0xfaa1 bytes) 000408 1788269932.854161 trunc from 0x040000 to 0x000e1b (0x3f1e4 bytes) 000409 1788269932.861877 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xe1c): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe1c): Mirror 9: 0x2173179a (stale) Mirror 10: 0x2173179a (stale) Mirror 11: 0x2173179a (stale) Mirror 12: 0x2173179a (stale) Mirror 15: 0x2173179a (stale) Mirror 16: 0x2173179a (stale) CRC-32 checksum value for chunk [0, 0xe1c): Mirror 1: 0x2173179a Mirror 17: 0x2173179a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe1c: skipped 000412 1788269933.238247 MIRROR_RESYNC 8 mirrors 000413 1788269933.651969 read 0x000c32 thru 0x000e1b (0x1ea bytes) 000417 1788269933.655029 mapread 0x0009ff thru 0x000e1b (0x41d bytes) 000418 1788269933.657208 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) 000419 1788269933.803868 read 0x00070f thru 0x000e1b (0x70d bytes) 000422 1788269933.809461 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) 000423 1788269933.960870 trunc from 0x000e1c to 0x02f7ff (0x2e9e4 bytes) 000425 1788269933.998995 mapread 0x0280b2 thru 0x02d02b (0x4f7a bytes) 000427 1788269934.012427 write 0x000c7b thru 0x0094f2 (0x8878 bytes) 000430 1788269934.018665 trunc from 0x02f800 to 0x02c1ad (0x3652 bytes) 000432 1788269934.032994 mapwrite 0x036171 thru 0x03d0f7 (0x6f87 bytes) 000433 1788269934.079807 read 0x038837 thru 0x03d0f7 (0x48c1 bytes) 000435 1788269934.080866 mapwrite 0x03b004 thru 0x03ffff (0x4ffc bytes) 000436 1788269934.113835 mapread 0x03452a thru 0x0357fd (0x12d4 bytes) 000437 1788269934.114672 write 0x009f94 thru 0x019f19 (0xff86 bytes) 000438 1788269934.121594 write 0x00a044 thru 0x011710 (0x76cd bytes) 000441 1788269934.125456 MIRROR_RESYNC 6 mirrors 000442 1788269934.686903 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) 000447 1788269934.869642 write 0x00840e thru 0x010ea0 (0x8a93 bytes) 000448 1788269934.917969 mapread 0x010e31 thru 0x01b027 (0xa1f7 bytes) 000449 1788269934.926625 trunc from 0x040000 to 0x0160a8 (0x29f57 bytes) 000451 1788269934.958587 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x160a9): Mirror 12: 0x3fafddad (stale) Mirror 15: 0x3fafddad (stale) Mirror 16: 0x3fafddad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x160a9): Mirror 12: 0x3fafddad (stale) Mirror 15: 0x3fafddad (stale) Mirror 16: 0x3fafddad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x160a9: skipped 000453 1788269935.124945 mapread 0x011189 thru 0x0148cc (0x3744 bytes) 000454 1788269935.138112 read 0x00f8b8 thru 0x00fd94 (0x4dd bytes) 000455 1788269935.138772 mapwrite 0x03b555 thru 0x03ffff (0x4aab bytes) 000456 1788269935.173706 mapwrite 0x03372b thru 0x03ab87 (0x745d bytes) 000457 1788269935.216613 MIRROR_RESYNC 4 mirrors 000459 1788269935.682681 read 0x0030df thru 0x00a2c7 (0x71e9 bytes) 000460 1788269935.685889 write 0x01a788 thru 0x02836d (0xdbe6 bytes) 000461 1788269935.712158 mapread 0x006ae6 thru 0x010bbb (0xa0d6 bytes) 000462 1788269935.721008 write 0x035683 thru 0x03ffff (0xa97d bytes) 000464 1788269935.724551 trunc from 0x040000 to 0x015d87 (0x2a278 bytes) 000465 1788269935.756088 read 0x01490f thru 0x015d87 (0x1479 bytes) 000466 1788269935.761627 trunc from 0x015d88 to 0x034a5d (0x1ecd6 bytes) 000467 1788269935.770080 write 0x039da8 thru 0x03eab2 (0x4d0b bytes) 000470 1788269935.772530 read 0x00f6f8 thru 0x01e3ff (0xed08 bytes) 000471 1788269935.780068 read 0x03e2ab thru 0x03eab2 (0x808 bytes) 000472 1788269935.780725 read 0x00fe6c thru 0x01464a (0x47df bytes) 000473 1788269935.783822 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eab3): Mirror 12: 0x608f8f2b (stale) Mirror 16: 0x608f8f2b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eab3): Mirror 12: 0x608f8f2b (stale) Mirror 16: 0x608f8f2b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eab3: skipped 000475 1788269935.963410 MIRROR_RESYNC 3 mirrors 000478 1788269936.356835 trunc from 0x03eab3 to 0x03bb75 (0x2f3d bytes) 000479 1788269936.386933 write 0x005de4 thru 0x0159a9 (0xfbc6 bytes) 000480 1788269936.395891 read 0x01671e thru 0x01f7e2 (0x90c5 bytes) 000481 1788269936.401379 trunc from 0x03bb76 to 0x0235a7 (0x185ce bytes) 000482 1788269936.414694 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x235a8): Mirror 12: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x235a8): Mirror 12: 0x2e926128 (stale) Mirror 16: 0x2e926128 (stale) CRC-32 checksum value for chunk [0, 0x235a8): Mirror 1: 0x7a82e576 Mirror 17: 0x7a82e576 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x235a8: skipped 000485 1788269936.824269 mapwrite 0x0009d5 thru 0x00130c (0x938 bytes) 000487 1788269936.883815 read 0x01cd6b thru 0x01ed36 (0x1fcc bytes) 000488 1788269936.886560 read 0x01e444 thru 0x020811 (0x23ce bytes) 000489 1788269936.891720 write 0x020fdd thru 0x02fb7b (0xeb9f bytes) 000493 1788269936.894075 trunc from 0x02fb7c to 0x033b9f (0x4024 bytes) 000494 1788269936.906784 read 0x009e57 thru 0x00a594 (0x73e bytes) 000497 1788269936.910311 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33ba0): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ba0): Mirror 12: 0xf873d7b6 (stale) Mirror 16: 0xf873d7b6 (stale) Mirror 17: 0x6d4723d3 (stale) CRC-32 checksum value for chunk [0, 0x33ba0): Mirror 1: 0x36dc3eb5 Mirror 18: 0x36dc3eb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ba0: skipped 000498 1788269937.472226 read 0x023f07 thru 0x026d17 (0x2e11 bytes) 000500 1788269937.491282 read 0x01d1ce thru 0x01e9bc (0x17ef bytes) 000501 1788269937.517494 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33ba0): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33ba0): Mirror 12: 0xf873d7b6 (stale) Mirror 16: 0xf873d7b6 (stale) Mirror 17: 0x6d4723d3 (stale) CRC-32 checksum value for chunk [0, 0x33ba0): Mirror 1: 0x36dc3eb5 Mirror 18: 0x36dc3eb5 Mirror 19: 0x36dc3eb5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33ba0: skipped 000502 1788269938.028117 mapread 0x004ffd thru 0x00cf6f (0x7f73 bytes) 000503 1788269938.046145 mapread 0x0291a6 thru 0x02aa38 (0x1893 bytes) 000504 1788269938.055744 mapwrite 0x03d4dd thru 0x03ffff (0x2b23 bytes) 000505 1788269938.181016 MIRROR_RESYNC 6 mirrors 000506 1788269939.331076 write 0x032803 thru 0x0378b9 (0x50b7 bytes) 000507 1788269939.383266 mapread 0x02e859 thru 0x02fd79 (0x1521 bytes) 000508 1788269939.394757 trunc from 0x040000 to 0x00d53c (0x32ac3 bytes) 000510 1788269939.414622 mapwrite 0x001357 thru 0x003aa7 (0x2751 bytes) 000511 1788269939.442647 mapwrite 0x01b09d thru 0x0285cb (0xd52f bytes) 000516 1788269939.509105 trunc from 0x0285cc to 0x018d92 (0xf839 bytes) 000520 1788269939.518480 read 0x004532 thru 0x0053bb (0xe8a bytes) 000521 1788269939.519534 mapwrite 0x00cfc4 thru 0x018a40 (0xba7d bytes) 000522 1788269939.556607 mapread 0x01658a thru 0x018d92 (0x2809 bytes) 000525 1788269939.557966 mapread 0x00a896 thru 0x018412 (0xdb7d bytes) 000526 1788269939.558892 write 0x029688 thru 0x035109 (0xba82 bytes) 000527 1788269939.560548 read 0x01861a thru 0x01a1aa (0x1b91 bytes) 000528 1788269939.571244 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3510a): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3510a): Mirror 12: 0x1194f7fe (stale) Mirror 16: 0x1194f7fe (stale) Mirror 17: 0x1194f7fe (stale) Mirror 18: 0x1194f7fe (stale) Mirror 19: 0x1194f7fe (stale) CRC-32 checksum value for chunk [0, 0x3510a): Mirror 1: 0x2c4b92d Mirror 20: 0x2c4b92d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3510a: skipped 000531 1788269940.135305 mapread 0x03058d thru 0x035109 (0x4b7d bytes) 000533 1788269940.159562 mapwrite 0x022fc8 thru 0x02bb62 (0x8b9b bytes) 000535 1788269940.251383 write 0x003be9 thru 0x0040b8 (0x4d0 bytes) 000536 1788269940.257245 mapread 0x0195a0 thru 0x02882e (0xf28f bytes) 000538 1788269940.264845 read 0x01d51b thru 0x021a3b (0x4521 bytes) 000539 1788269940.269074 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3510a): Mirror 12: 0x1194f7fe (stale) Mirror 16: 0x1194f7fe (stale) Mirror 17: 0x1194f7fe (stale) Mirror 18: 0x1194f7fe (stale) Mirror 20: 0x2c4b92d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3510a): Mirror 12: 0x1194f7fe (stale) Mirror 16: 0x1194f7fe (stale) Mirror 17: 0x1194f7fe (stale) Mirror 18: 0x1194f7fe (stale) Mirror 20: 0x2c4b92d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3510a: skipped 000540 1788269940.491954 mapwrite 0x00bef9 thru 0x01b4bb (0xf5c3 bytes) 000541 1788269940.529885 read 0x009dbe thru 0x009fa9 (0x1ec bytes) 000542 1788269940.531285 mapwrite 0x010b7f thru 0x01432a (0x37ac bytes) 000543 1788269940.546730 trunc from 0x03510a to 0x017b91 (0x1d578 bytes) 000544 1788269940.565130 mapread 0x015468 thru 0x017b91 (0x272a bytes) 000545 1788269940.568104 MIRROR_RESYNC 6 mirrors 000546 1788269940.968111 read 0x0113b5 thru 0x017b91 (0x67dd bytes) 000548 1788269940.974497 MIRROR_EXTEND to 7 mirrors 000552 1788269941.315797 trunc from 0x017b92 to 0x00827d (0xf914 bytes) 000555 1788269941.352706 mapwrite 0x02b624 thru 0x035b0f (0xa4ec bytes) 000556 1788269941.391558 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35b10): Mirror 12: 0x1c054823 (stale) Mirror 16: 0x1c054823 (stale) Mirror 17: 0x1c054823 (stale) Mirror 18: 0x1c054823 (stale) Mirror 21: 0x1c054823 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35b10): Mirror 12: 0x1c054823 (stale) Mirror 16: 0x1c054823 (stale) Mirror 17: 0x1c054823 (stale) Mirror 18: 0x1c054823 (stale) Mirror 21: 0x1c054823 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35b10: skipped 000557 1788269941.541364 read 0x02760b thru 0x032931 (0xb327 bytes) 000559 1788269941.546406 mapwrite 0x0338de thru 0x03a0a7 (0x67ca bytes) 000561 1788269941.580621 write 0x0369bd thru 0x03ffff (0x9643 bytes) 000562 1788269941.583691 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5f67552b (stale) Mirror 16: 0x5f67552b (stale) Mirror 17: 0x5f67552b (stale) Mirror 21: 0x5f67552b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5f67552b (stale) Mirror 16: 0x5f67552b (stale) Mirror 17: 0x5f67552b (stale) Mirror 21: 0x5f67552b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000564 1788269941.761585 write 0x03320a thru 0x03ffff (0xcdf6 bytes) 000566 1788269941.765869 mapwrite 0x0208cf thru 0x023a99 (0x31cb bytes) 000567 1788269941.803166 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x5f67552b (stale) Mirror 16: 0x5f67552b (stale) Mirror 17: 0x5f67552b (stale) Mirror 21: 0x5f67552b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x304d0bd3 Mirror 22: 0x304d0bd3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000568 1788269942.331026 mapread 0x008dac thru 0x0144e8 (0xb73d bytes) 000569 1788269942.347335 read 0x02dace thru 0x03187c (0x3daf bytes) 000572 1788269942.362413 read 0x021a9f thru 0x027ca7 (0x6209 bytes) 000573 1788269942.370198 write 0x030118 thru 0x03e5ab (0xe494 bytes) 000574 1788269942.401691 read 0x02d128 thru 0x034c12 (0x7aeb bytes) 000575 1788269942.406504 write 0x032d84 thru 0x036329 (0x35a6 bytes) 000576 1788269942.407426 trunc from 0x040000 to 0x02d68a (0x12975 bytes) 000577 1788269942.417020 mapread 0x00ca7d thru 0x012ff4 (0x6578 bytes) 000578 1788269942.418070 trunc from 0x02d68b to 0x03821a (0xab90 bytes) 000580 1788269942.429616 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3821b): Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3821b): Mirror 12: 0x155effcd (stale) Mirror 16: 0x155effcd (stale) Mirror 17: 0x155effcd (stale) Mirror 21: 0x155effcd (stale) Mirror 22: 0xcd21e29f (stale) CRC-32 checksum value for chunk [0, 0x3821b): Mirror 1: 0xe2171e7f Mirror 23: 0xe2171e7f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3821b: skipped 000581 1788269942.974248 mapread 0x016ecd thru 0x01c129 (0x525d bytes) 000582 1788269942.988492 write 0x037c51 thru 0x03ffff (0x83af bytes) 000585 1788269943.024234 write 0x03e12b thru 0x03ffff (0x1ed5 bytes) 000586 1788269943.027690 MIRROR_RESYNC 7 mirrors 000587 1788269944.306059 mapwrite 0x02b3eb thru 0x0308c2 (0x54d8 bytes) 000588 1788269944.370318 write 0x0327e2 thru 0x038436 (0x5c55 bytes) 000590 1788269944.373798 mapread 0x00a35f thru 0x01439d (0xa03f bytes) 000591 1788269944.383579 mapwrite 0x005222 thru 0x011810 (0xc5ef bytes) 000593 1788269944.404040 mapwrite 0x012a39 thru 0x020a37 (0xdfff bytes) 000594 1788269944.428564 MIRROR_RESYNC 7 mirrors 000595 1788269945.118667 mapwrite 0x00c4cc thru 0x017bcc (0xb701 bytes) 000596 1788269945.209208 trunc from 0x040000 to 0x0210ed (0x1ef12 bytes) 000597 1788269945.225273 mapwrite 0x0221cf thru 0x031282 (0xf0b4 bytes) 000598 1788269945.269466 trunc from 0x031283 to 0x00c8a4 (0x249de bytes) 000601 1788269945.280895 trunc from 0x00c8a5 to 0x020a24 (0x14180 bytes) 000602 1788269945.301124 MIRROR_RESYNC 7 mirrors 000605 1788269945.966051 mapwrite 0x0027b7 thru 0x0067de (0x4028 bytes) 000606 1788269946.069782 read 0x011a77 thru 0x020a24 (0xefae bytes) 000607 1788269946.083387 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20a25): Mirror 12: 0xfbd4db72 (stale) Mirror 17: 0xfbd4db72 (stale) Mirror 21: 0xfbd4db72 (stale) Mirror 22: 0xfbd4db72 (stale) Mirror 23: 0xfbd4db72 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20a25): Mirror 12: 0xfbd4db72 (stale) Mirror 17: 0xfbd4db72 (stale) Mirror 21: 0xfbd4db72 (stale) Mirror 22: 0xfbd4db72 (stale) Mirror 23: 0xfbd4db72 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20a25: skipped truncating to largest ever: 0x3ff10 000608 1788269946.411501 trunc from 0x020a25 to 0x03ff0f (0x1f4eb bytes) 000610 1788269946.434783 mapread 0x0109a7 thru 0x0193b7 (0x8a11 bytes) 000611 1788269946.454484 mapwrite 0x02308e thru 0x02544d (0x23c0 bytes) 000613 1788269946.499992 mapwrite 0x03b96c thru 0x03db4f (0x21e4 bytes) 000615 1788269946.534744 read 0x03d791 thru 0x03ff0f (0x277f bytes) 000618 1788269946.536310 mapwrite 0x0397f2 thru 0x03d606 (0x3e15 bytes) 000619 1788269946.565792 mapwrite 0x002c0f thru 0x0094a3 (0x6895 bytes) 000620 1788269946.599159 trunc from 0x03ff10 to 0x0377e1 (0x872e bytes) 000621 1788269946.612230 write 0x039148 thru 0x03ffff (0x6eb8 bytes) 000622 1788269946.613575 write 0x00c621 thru 0x012a68 (0x6448 bytes) 000623 1788269946.614480 read 0x00df35 thru 0x018d98 (0xae64 bytes) 000625 1788269946.615451 mapwrite 0x027d28 thru 0x034aab (0xcd84 bytes) 000627 1788269946.654040 MIRROR_RESYNC 6 mirrors 000628 1788269948.042368 trunc from 0x040000 to 0x00e6eb (0x31914 bytes) 000629 1788269948.067389 read 0x00d74a thru 0x00e6eb (0xfa2 bytes) 000630 1788269948.072224 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xe6ec): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe6ec): Mirror 12: 0xf227c17b (stale) Mirror 17: 0xf227c17b (stale) Mirror 21: 0xf227c17b (stale) Mirror 22: 0xf227c17b (stale) Mirror 23: 0xf227c17b (stale) CRC-32 checksum value for chunk [0, 0xe6ec): Mirror 1: 0xf227c17b Mirror 24: 0xf227c17b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe6ec: skipped 000631 1788269948.557950 read 0x0026b1 thru 0x00b501 (0x8e51 bytes) 000632 1788269948.563936 write 0x03460d thru 0x03e372 (0x9d66 bytes) 000634 1788269948.601895 write 0x0212da thru 0x02e552 (0xd279 bytes) 000638 1788269948.608777 mapwrite 0x014888 thru 0x01708d (0x2806 bytes) 000640 1788269948.640323 trunc from 0x03e373 to 0x01d79c (0x20bd6 bytes) 000641 1788269948.652980 write 0x0003f8 thru 0x005a73 (0x567c bytes) 000643 1788269948.657955 mapwrite 0x01c5c6 thru 0x024ea3 (0x88de bytes) 000644 1788269948.698233 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24ea4): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24ea4): Mirror 12: 0xbbaf2eb0 (stale) Mirror 17: 0xbbaf2eb0 (stale) Mirror 21: 0xbbaf2eb0 (stale) Mirror 22: 0xbbaf2eb0 (stale) Mirror 23: 0xbbaf2eb0 (stale) Mirror 24: 0x8191b755 (stale) CRC-32 checksum value for chunk [0, 0x24ea4): Mirror 1: 0xa49dbf58 Mirror 25: 0xa49dbf58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24ea4: skipped 000645 1788269949.223591 mapwrite 0x03f2dc thru 0x03ffff (0xd24 bytes) 000649 1788269949.375994 read 0x026a62 thru 0x02ea08 (0x7fa7 bytes) 000650 1788269949.385156 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x876ca2d7 (stale) Mirror 17: 0x876ca2d7 (stale) Mirror 21: 0x876ca2d7 (stale) Mirror 22: 0x876ca2d7 (stale) Mirror 23: 0x876ca2d7 (stale) Mirror 24: 0x17c13b92 (stale) Mirror 25: 0xee66be76 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xef76f900 Mirror 26: 0xef76f900 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000651 1788269949.923213 mapread 0x00f4f1 thru 0x01508a (0x5b9a bytes) 000652 1788269949.939569 write 0x00dd06 thru 0x011af8 (0x3df3 bytes) 000654 1788269949.999079 write 0x004c45 thru 0x013a2b (0xede7 bytes) 000658 1788269950.012939 write 0x01a57c thru 0x026760 (0xc1e5 bytes) 000660 1788269950.027874 read 0x03caa9 thru 0x03ffff (0x3557 bytes) 000661 1788269950.037154 trunc from 0x040000 to 0x00d7fa (0x32805 bytes) 000662 1788269950.085100 write 0x011c63 thru 0x01c94a (0xace8 bytes) 000663 1788269950.087113 read 0x001329 thru 0x001b12 (0x7ea bytes) 000665 1788269950.100015 write 0x0065ac thru 0x00759b (0xff0 bytes) 000667 1788269950.101430 read 0x0097a3 thru 0x00ce1e (0x367c bytes) 000668 1788269950.101904 mapread 0x000d64 thru 0x002b10 (0x1dad bytes) 000670 1788269950.103523 write 0x035556 thru 0x03ffe1 (0xaa8c bytes) 000671 1788269950.104971 read 0x020f16 thru 0x0288a5 (0x7990 bytes) 000673 1788269950.120657 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ffe2): Mirror 12: 0x8c0df967 (stale) Mirror 17: 0x8c0df967 (stale) Mirror 21: 0x8c0df967 (stale) Mirror 22: 0x8c0df967 (stale) Mirror 23: 0x8c0df967 (stale) Mirror 25: 0x7df34a29 (stale) Mirror 26: 0x9e8d877d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ffe2): Mirror 12: 0x8c0df967 (stale) Mirror 17: 0x8c0df967 (stale) Mirror 21: 0x8c0df967 (stale) Mirror 22: 0x8c0df967 (stale) Mirror 23: 0x8c0df967 (stale) Mirror 25: 0x7df34a29 (stale) Mirror 26: 0x9e8d877d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ffe2: skipped 000675 1788269950.494587 trunc from 0x03ffe2 to 0x025ff5 (0x19fec bytes) 000676 1788269950.504598 mapwrite 0x02e76e thru 0x02e771 (0x4 bytes) 000677 1788269950.553527 write 0x03850a thru 0x03ffff (0x7af6 bytes) 000678 1788269950.555039 mapwrite 0x03c237 thru 0x03ffff (0x3dc9 bytes) 000679 1788269950.596274 read 0x0057a8 thru 0x012438 (0xcc91 bytes) 000680 1788269950.601562 mapread 0x032f87 thru 0x03ffff (0xd079 bytes) 000681 1788269950.617842 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x876ca2d7 (stale) Mirror 17: 0x876ca2d7 (stale) Mirror 21: 0x876ca2d7 (stale) Mirror 22: 0x876ca2d7 (stale) Mirror 23: 0x876ca2d7 (stale) Mirror 25: 0xee66be76 (stale) Mirror 26: 0xef76f900 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x95df13f6 Mirror 27: 0x95df13f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000686 1788269951.625616 read 0x02d07c thru 0x033f2f (0x6eb4 bytes) 000687 1788269951.637704 mapread 0x03a99e thru 0x03ffff (0x5662 bytes) 000688 1788269951.667709 write 0x02e90d thru 0x038604 (0x9cf8 bytes) 000690 1788269951.755159 trunc from 0x040000 to 0x022dc2 (0x1d23d bytes) 000691 1788269951.766817 mapread 0x018f42 thru 0x01da1c (0x4adb bytes) 000692 1788269951.775515 write 0x00ec72 thru 0x016047 (0x73d6 bytes) 000694 1788269951.782562 read 0x00dbc5 thru 0x01521f (0x765b bytes) 000695 1788269951.791769 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x22dc3): Mirror 12: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22dc3): Mirror 12: 0xa5b995ca (stale) Mirror 17: 0xa5b995ca (stale) Mirror 21: 0xa5b995ca (stale) Mirror 22: 0xa5b995ca (stale) Mirror 23: 0xa5b995ca (stale) Mirror 25: 0x6534a03a (stale) Mirror 26: 0x6534a03a (stale) Mirror 27: 0x3204571e (stale) CRC-32 checksum value for chunk [0, 0x22dc3): Mirror 1: 0x301158c7 Mirror 28: 0x301158c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22dc3: skipped 000696 1788269952.709412 read 0x013034 thru 0x01f64e (0xc61b bytes) 000697 1788269952.722721 mapwrite 0x0382e9 thru 0x03b470 (0x3188 bytes) 000698 1788269952.853597 mapread 0x025496 thru 0x02b280 (0x5deb bytes) 000699 1788269952.860214 mapread 0x03a0ca thru 0x03b470 (0x13a7 bytes) 000700 1788269952.861123 trunc from 0x03b471 to 0x0055d3 (0x35e9d bytes) 000703 1788269952.875376 mapread 0x00235a thru 0x0055d3 (0x327a bytes) 000704 1788269952.881313 trunc from 0x0055d4 to 0x01d3de (0x17e0b bytes) 000705 1788269952.895599 MIRROR_RESYNC 10 mirrors 000706 1788269954.418562 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) 000707 1788269954.757162 write 0x013bf3 thru 0x014cc5 (0x10d3 bytes) 000709 1788269954.875742 read 0x00ad62 thru 0x00b8d2 (0xb71 bytes) 000710 1788269954.895928 MIRROR_RESYNC 9 mirrors 000712 1788269956.970094 trunc from 0x01d3df to 0x005693 (0x17d4b bytes) 000713 1788269957.051480 write 0x03ac88 thru 0x03ffff (0x5378 bytes) 000715 1788269957.055459 read 0x034258 thru 0x035d98 (0x1b41 bytes) 000716 1788269957.064933 mapwrite 0x022808 thru 0x02f1be (0xc9b7 bytes) 000717 1788269957.122454 MIRROR_RESYNC 9 mirrors 000719 1788269959.064545 mapwrite 0x03c601 thru 0x03ffff (0x39ff bytes) 000722 1788269959.159629 trunc from 0x040000 to 0x02430b (0x1bcf4 bytes) 000723 1788269959.171580 read 0x0076e5 thru 0x014f07 (0xd823 bytes) 000725 1788269959.178949 MIRROR_RESYNC 9 mirrors 000726 1788269960.296295 read 0x0055fb thru 0x00ddbe (0x87c4 bytes) 000727 1788269960.302011 read 0x012f67 thru 0x01eb47 (0xbbe1 bytes) 000728 1788269960.310630 MIRROR_RESYNC 9 mirrors 000729 1788269960.764110 mapwrite 0x008696 thru 0x016265 (0xdbd0 bytes) 000730 1788269960.887275 write 0x01f8f3 thru 0x02ad47 (0xb455 bytes) 000731 1788269960.895461 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2ad48): Mirror 12: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad48): Mirror 12: 0x5483672f (stale) Mirror 21: 0x5483672f (stale) Mirror 22: 0x5483672f (stale) Mirror 23: 0x5483672f (stale) Mirror 25: 0x5483672f (stale) Mirror 26: 0x5483672f (stale) Mirror 27: 0x5483672f (stale) Mirror 28: 0x5483672f (stale) CRC-32 checksum value for chunk [0, 0x2ad48): Mirror 1: 0xab38785d Mirror 29: 0xab38785d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad48: skipped 000732 1788269961.824682 mapwrite 0x03a584 thru 0x03f5ec (0x5069 bytes) 000733 1788269961.952796 write 0x001281 thru 0x005633 (0x43b3 bytes) 000735 1788269961.976531 write 0x023289 thru 0x024ba9 (0x1921 bytes) 000736 1788269961.992107 mapread 0x02ca8e thru 0x02ee22 (0x2395 bytes) 000737 1788269962.009923 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3f5ed): Mirror 12: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f5ed): Mirror 12: 0xe5c742be (stale) Mirror 21: 0xe5c742be (stale) Mirror 22: 0xe5c742be (stale) Mirror 23: 0xe5c742be (stale) Mirror 25: 0xe5c742be (stale) Mirror 26: 0xe5c742be (stale) Mirror 27: 0xe5c742be (stale) Mirror 28: 0xe5c742be (stale) Mirror 29: 0x8280116d (stale) CRC-32 checksum value for chunk [0, 0x3f5ed): Mirror 1: 0x72b4f225 Mirror 30: 0x72b4f225 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f5ed: skipped 000741 1788269963.086025 write 0x00eebe thru 0x00ff86 (0x10c9 bytes) 000742 1788269963.171644 write 0x02006e thru 0x021f59 (0x1eec bytes) 000743 1788269963.183522 MIRROR_RESYNC 11 mirrors 000746 1788269966.133950 write 0x00bb5e thru 0x00bb9d (0x40 bytes) 000747 1788269966.183968 trunc from 0x03f5ed to 0x01e635 (0x20fb7 bytes) 000751 1788269966.205076 mapread 0x019358 thru 0x01e635 (0x52de bytes) 000755 1788269966.221129 mapwrite 0x032cfa thru 0x035d2a (0x3031 bytes) 000756 1788269966.287560 read 0x014fe3 thru 0x019abf (0x4add bytes) 000758 1788269966.288696 write 0x02fdf3 thru 0x03c153 (0xc361 bytes) 000761 1788269966.293720 mapwrite 0x02a62a thru 0x02d84a (0x3221 bytes) 000762 1788269966.336694 trunc from 0x03c154 to 0x01cf94 (0x1f1bf bytes) 000763 1788269966.354705 read 0x0122cb thru 0x019c14 (0x794a bytes) 000764 1788269966.366387 mapread 0x0173c8 thru 0x017bfd (0x836 bytes) 000765 1788269966.367327 read 0x016371 thru 0x01cc97 (0x6927 bytes) 000766 1788269966.367703 mapread 0x00a83a thru 0x013bbe (0x9385 bytes) 000768 1788269966.369084 trunc from 0x01cf95 to 0x015247 (0x7d4d bytes) 000769 1788269966.392236 read 0x006e2e thru 0x013d65 (0xcf38 bytes) 000770 1788269966.392996 mapwrite 0x019de0 thru 0x01ca42 (0x2c63 bytes) 000772 1788269966.442639 write 0x026c4b thru 0x0273b4 (0x76a bytes) 000773 1788269966.448218 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x273b5): Mirror 12: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x273b5): Mirror 12: 0x46a9ebef (stale) Mirror 21: 0x46a9ebef (stale) Mirror 22: 0x46a9ebef (stale) Mirror 23: 0x46a9ebef (stale) Mirror 25: 0x46a9ebef (stale) Mirror 26: 0x46a9ebef (stale) Mirror 27: 0x46a9ebef (stale) Mirror 28: 0x46a9ebef (stale) Mirror 29: 0x46a9ebef (stale) Mirror 30: 0x46a9ebef (stale) CRC-32 checksum value for chunk [0, 0x273b5): Mirror 1: 0xc215ddbb Mirror 31: 0xc215ddbb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x273b5: skipped 000776 1788269967.577033 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x273b5): Mirror 12: 0x46a9ebef (stale) Mirror 21: 0x46a9ebef (stale) Mirror 22: 0x46a9ebef (stale) Mirror 23: 0x46a9ebef (stale) Mirror 25: 0x46a9ebef (stale) Mirror 26: 0x46a9ebef (stale) Mirror 27: 0x46a9ebef (stale) Mirror 29: 0x46a9ebef (stale) Mirror 30: 0x46a9ebef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x273b5): Mirror 12: 0x46a9ebef (stale) Mirror 21: 0x46a9ebef (stale) Mirror 22: 0x46a9ebef (stale) Mirror 23: 0x46a9ebef (stale) Mirror 25: 0x46a9ebef (stale) Mirror 26: 0x46a9ebef (stale) Mirror 27: 0x46a9ebef (stale) Mirror 29: 0x46a9ebef (stale) Mirror 30: 0x46a9ebef (stale) CRC-32 checksum value for chunk [0, 0x273b5): Mirror 1: 0xc215ddbb Mirror 31: 0xc215ddbb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x273b5: skipped 000778 1788269968.426210 trunc from 0x0273b5 to 0x00f39e (0x18016 bytes) 000781 1788269968.506657 write 0x038e22 thru 0x03ffff (0x71de bytes) 000782 1788269968.508475 mapwrite 0x0122e4 thru 0x01fead (0xdbca bytes) 000783 1788269968.591996 trunc from 0x040000 to 0x028563 (0x17a9c bytes) 000786 1788269968.606122 mapread 0x021ae4 thru 0x02325b (0x1778 bytes) 000787 1788269968.607061 trunc from 0x028564 to 0x03a2ca (0x11d67 bytes) 000789 1788269968.623638 MIRROR_RESYNC 11 mirrors 000790 1788269971.242405 mapwrite 0x011b49 thru 0x01a650 (0x8b08 bytes) 000791 1788269971.351740 mapwrite 0x0005ae thru 0x00f75c (0xf1af bytes) 000794 1788269971.382713 mapwrite 0x01e5e9 thru 0x0296e3 (0xb0fb bytes) 000795 1788269971.422847 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3a2cb): Mirror 12: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2cb): Mirror 12: 0x7c6d315a (stale) Mirror 21: 0x7c6d315a (stale) Mirror 22: 0x7c6d315a (stale) Mirror 23: 0x7c6d315a (stale) Mirror 25: 0x7c6d315a (stale) Mirror 26: 0x7c6d315a (stale) Mirror 27: 0x7c6d315a (stale) Mirror 29: 0x7c6d315a (stale) Mirror 30: 0x7c6d315a (stale) Mirror 31: 0x7c6d315a (stale) CRC-32 checksum value for chunk [0, 0x3a2cb): Mirror 1: 0x3752cc3 Mirror 32: 0x3752cc3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2cb: skipped 000796 1788269971.984365 mapread 0x030384 thru 0x03a2ca (0x9f47 bytes) 000798 1788269972.012787 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a2cb): Mirror 12: 0x7c6d315a (stale) Mirror 21: 0x7c6d315a (stale) Mirror 22: 0x7c6d315a (stale) Mirror 23: 0x7c6d315a (stale) Mirror 25: 0x7c6d315a (stale) Mirror 26: 0x7c6d315a (stale) Mirror 27: 0x7c6d315a (stale) Mirror 30: 0x7c6d315a (stale) Mirror 31: 0x7c6d315a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2cb): Mirror 12: 0x7c6d315a (stale) Mirror 21: 0x7c6d315a (stale) Mirror 22: 0x7c6d315a (stale) Mirror 23: 0x7c6d315a (stale) Mirror 25: 0x7c6d315a (stale) Mirror 26: 0x7c6d315a (stale) Mirror 27: 0x7c6d315a (stale) Mirror 30: 0x7c6d315a (stale) Mirror 31: 0x7c6d315a (stale) CRC-32 checksum value for chunk [0, 0x3a2cb): Mirror 1: 0x3752cc3 Mirror 32: 0x3752cc3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2cb: skipped 000799 1788269972.553985 trunc from 0x03a2cb to 0x017e6e (0x2245c bytes) 000801 1788269972.599394 write 0x021028 thru 0x0256a9 (0x4682 bytes) 000803 1788269972.600971 trunc from 0x0256aa to 0x02ead3 (0x942a bytes) 000805 1788269972.611756 write 0x010821 thru 0x018e2e (0x860e bytes) 000806 1788269972.622436 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2ead4): Mirror 12: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ead4): Mirror 12: 0xa165ad72 (stale) Mirror 21: 0xa165ad72 (stale) Mirror 22: 0xa165ad72 (stale) Mirror 23: 0xa165ad72 (stale) Mirror 25: 0xa165ad72 (stale) Mirror 26: 0xa165ad72 (stale) Mirror 27: 0xa165ad72 (stale) Mirror 30: 0xa165ad72 (stale) Mirror 31: 0xa165ad72 (stale) Mirror 32: 0x2ef187bd (stale) CRC-32 checksum value for chunk [0, 0x2ead4): Mirror 1: 0x492eb123 Mirror 33: 0x492eb123 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ead4: skipped 000808 1788269973.214887 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x2ead4): Mirror 12: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ead4): Mirror 12: 0xa165ad72 (stale) Mirror 21: 0xa165ad72 (stale) Mirror 22: 0xa165ad72 (stale) Mirror 23: 0xa165ad72 (stale) Mirror 25: 0xa165ad72 (stale) Mirror 26: 0xa165ad72 (stale) Mirror 27: 0xa165ad72 (stale) Mirror 30: 0xa165ad72 (stale) Mirror 31: 0xa165ad72 (stale) Mirror 32: 0x2ef187bd (stale) CRC-32 checksum value for chunk [0, 0x2ead4): Mirror 1: 0x492eb123 Mirror 33: 0x492eb123 Mirror 34: 0x492eb123 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ead4: skipped 000810 1788269973.742467 trunc from 0x02ead4 to 0x021942 (0xd191 bytes) 000812 1788269973.857701 read 0x0079cd thru 0x00c95a (0x4f8e bytes) 000814 1788269973.863450 read 0x019817 thru 0x021942 (0x812c bytes) 000815 1788269973.871597 mapwrite 0x02268c thru 0x02cd59 (0xa6ce bytes) 000817 1788269973.959809 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cd5a): Mirror 12: 0x1b4b54db (stale) Mirror 21: 0x1b4b54db (stale) Mirror 22: 0x1b4b54db (stale) Mirror 23: 0x1b4b54db (stale) Mirror 25: 0x1b4b54db (stale) Mirror 26: 0x1b4b54db (stale) Mirror 27: 0x1b4b54db (stale) Mirror 30: 0x1b4b54db (stale) Mirror 31: 0x1b4b54db (stale) Mirror 32: 0x5578cb52 (stale) Mirror 33: 0xff3432c1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd5a): Mirror 12: 0x1b4b54db (stale) Mirror 21: 0x1b4b54db (stale) Mirror 22: 0x1b4b54db (stale) Mirror 23: 0x1b4b54db (stale) Mirror 25: 0x1b4b54db (stale) Mirror 26: 0x1b4b54db (stale) Mirror 27: 0x1b4b54db (stale) Mirror 30: 0x1b4b54db (stale) Mirror 31: 0x1b4b54db (stale) Mirror 32: 0x5578cb52 (stale) Mirror 33: 0xff3432c1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd5a: skipped 000818 1788269974.452261 mapwrite 0x018a39 thru 0x02571a (0xcce2 bytes) 000819 1788269974.521711 write 0x022cf3 thru 0x02f58e (0xc89c bytes) 000820 1788269974.523479 mapwrite 0x0344ed thru 0x03db1c (0x9630 bytes) 000821 1788269974.605164 mapread 0x00ddfb thru 0x0192e8 (0xb4ee bytes) 000822 1788269974.614536 mapread 0x0006ea thru 0x000ecd (0x7e4 bytes) 000823 1788269974.628927 mapread 0x013db6 thru 0x01cf01 (0x914c bytes) 000824 1788269974.629992 trunc from 0x03db1d to 0x0360f2 (0x7a2a bytes) 000827 1788269974.660056 mapwrite 0x02b826 thru 0x0323b0 (0x6b8b bytes) 000828 1788269974.700416 mapwrite 0x0021db thru 0x010c7f (0xeaa5 bytes) 000829 1788269974.738329 MIRROR_RESYNC 12 mirrors 000830 1788269975.989504 MIRROR_RESYNC 12 mirrors 000831 1788269976.458265 trunc from 0x0360f3 to 0x030518 (0x5bda bytes) 000832 1788269976.521250 trunc from 0x030519 to 0x016d03 (0x19815 bytes) 000833 1788269976.543254 trunc from 0x016d04 to 0x01308e (0x3c75 bytes) 000834 1788269976.555738 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1308f): Mirror 12: 0x58542c2 (stale) Mirror 21: 0x58542c2 (stale) Mirror 22: 0x58542c2 (stale) Mirror 23: 0x58542c2 (stale) Mirror 25: 0x58542c2 (stale) Mirror 27: 0x58542c2 (stale) Mirror 30: 0x58542c2 (stale) Mirror 31: 0x58542c2 (stale) Mirror 32: 0x58542c2 (stale) Mirror 33: 0x58542c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1308f): Mirror 12: 0x58542c2 (stale) Mirror 21: 0x58542c2 (stale) Mirror 22: 0x58542c2 (stale) Mirror 23: 0x58542c2 (stale) Mirror 25: 0x58542c2 (stale) Mirror 27: 0x58542c2 (stale) Mirror 30: 0x58542c2 (stale) Mirror 31: 0x58542c2 (stale) Mirror 32: 0x58542c2 (stale) Mirror 33: 0x58542c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1308f: skipped 000837 1788269976.951948 trunc from 0x01308f to 0x00a5b6 (0x8ad8 bytes) 000839 1788269976.970025 mapread 0x0036ee thru 0x003c62 (0x575 bytes) 000840 1788269976.973467 mapwrite 0x02f96d thru 0x034902 (0x4f96 bytes) 000842 1788269977.047777 mapread 0x000ab5 thru 0x003b41 (0x308d bytes) 000844 1788269977.058070 MIRROR_RESYNC 11 mirrors 000846 1788269978.931692 write 0x00590c thru 0x0145e0 (0xecd5 bytes) 000847 1788269978.985272 read 0x011a38 thru 0x01e570 (0xcb39 bytes) 000849 1788269978.998366 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34903): Mirror 12: 0x2913a010 (stale) Mirror 21: 0x2913a010 (stale) Mirror 22: 0x2913a010 (stale) Mirror 23: 0x2913a010 (stale) Mirror 25: 0x2913a010 (stale) Mirror 30: 0x2913a010 (stale) Mirror 31: 0x2913a010 (stale) Mirror 32: 0x2913a010 (stale) Mirror 33: 0x2913a010 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34903): Mirror 12: 0x2913a010 (stale) Mirror 21: 0x2913a010 (stale) Mirror 22: 0x2913a010 (stale) Mirror 23: 0x2913a010 (stale) Mirror 25: 0x2913a010 (stale) Mirror 30: 0x2913a010 (stale) Mirror 31: 0x2913a010 (stale) Mirror 32: 0x2913a010 (stale) Mirror 33: 0x2913a010 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34903: skipped 000850 1788269979.433825 mapwrite 0x026403 thru 0x03344d (0xd04b bytes) 000852 1788269979.502326 mapwrite 0x0135fe thru 0x017d88 (0x478b bytes) 000853 1788269979.538990 write 0x016f94 thru 0x0244a5 (0xd512 bytes) 000855 1788269979.540381 mapread 0x0000be thru 0x000b75 (0xab8 bytes) 000858 1788269979.549152 read 0x02b9dc thru 0x034902 (0x8f27 bytes) 000859 1788269979.552471 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x34903): Mirror 12: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34903): Mirror 12: 0x2913a010 (stale) Mirror 21: 0x2913a010 (stale) Mirror 22: 0x2913a010 (stale) Mirror 23: 0x2913a010 (stale) Mirror 25: 0x2913a010 (stale) Mirror 30: 0x2913a010 (stale) Mirror 31: 0x2913a010 (stale) Mirror 32: 0x2913a010 (stale) Mirror 33: 0x2913a010 (stale) CRC-32 checksum value for chunk [0, 0x34903): Mirror 1: 0x3c2c60ec Mirror 34: 0x3c2c60ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34903: skipped 000860 1788269980.214260 trunc from 0x034903 to 0x003536 (0x313cc bytes) 000861 1788269980.284409 mapread 0x000436 thru 0x003536 (0x3101 bytes) 000864 1788269980.298481 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3537): Mirror 12: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3537): Mirror 12: 0x7aa7604 (stale) Mirror 21: 0x7aa7604 (stale) Mirror 22: 0x7aa7604 (stale) Mirror 23: 0x7aa7604 (stale) Mirror 25: 0x7aa7604 (stale) Mirror 30: 0x7aa7604 (stale) Mirror 31: 0x7aa7604 (stale) Mirror 32: 0x7aa7604 (stale) Mirror 33: 0x7aa7604 (stale) Mirror 34: 0x7aa7604 (stale) CRC-32 checksum value for chunk [0, 0x3537): Mirror 1: 0x7aa7604 Mirror 35: 0x7aa7604 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3537: skipped 000869 1788269980.890562 mapwrite 0x00a9c5 thru 0x010726 (0x5d62 bytes) 000870 1788269981.076841 trunc from 0x010727 to 0x004f88 (0xb79e bytes) 000872 1788269981.102313 trunc from 0x004f89 to 0x020d02 (0x1bd7a bytes) 000875 1788269981.126505 mapread 0x014351 thru 0x016810 (0x24c0 bytes) 000877 1788269981.142337 write 0x0340cf thru 0x03871c (0x464e bytes) 000878 1788269981.143551 trunc from 0x03871d to 0x031ac4 (0x6c58 bytes) 000880 1788269981.172750 write 0x00c023 thru 0x01b403 (0xf3e1 bytes) 000881 1788269981.180730 mapwrite 0x032ccd thru 0x03a8ad (0x7be1 bytes) 000882 1788269981.320338 MIRROR_RESYNC 12 mirrors 000883 1788269983.540725 read 0x000939 thru 0x007244 (0x690c bytes) 000884 1788269983.544858 trunc from 0x03a8ae to 0x0053b1 (0x354fc bytes) 000886 1788269983.581904 trunc from 0x0053b2 to 0x0311c8 (0x2be17 bytes) 000888 1788269983.594416 mapread 0x0255c2 thru 0x02a37c (0x4dbb bytes) 000890 1788269983.608020 write 0x03eef4 thru 0x03ffff (0x110c bytes) 000892 1788269983.609320 write 0x02bb87 thru 0x02fad2 (0x3f4c bytes) 000893 1788269983.610218 trunc from 0x040000 to 0x007849 (0x387b6 bytes) 000896 1788269983.618256 read 0x004548 thru 0x007849 (0x3302 bytes) 000898 1788269983.623300 read 0x0069d7 thru 0x007849 (0xe73 bytes) 000899 1788269983.623876 write 0x0270f4 thru 0x032b64 (0xba71 bytes) 000901 1788269983.625478 trunc from 0x032b65 to 0x008f97 (0x29bcd bytes) 000902 1788269983.633722 trunc from 0x008f98 to 0x022246 (0x192af bytes) 000904 1788269983.639694 write 0x03e98f thru 0x03ffff (0x1671 bytes) 000905 1788269983.640931 write 0x02e0ea thru 0x034fc7 (0x6ede bytes) 000906 1788269983.647272 read 0x01802f thru 0x01935a (0x132c bytes) 000907 1788269983.650437 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xe671bdec (stale) Mirror 21: 0xe671bdec (stale) Mirror 22: 0xe671bdec (stale) Mirror 23: 0xe671bdec (stale) Mirror 25: 0xe671bdec (stale) Mirror 30: 0xe671bdec (stale) Mirror 31: 0xe671bdec (stale) Mirror 32: 0xe671bdec (stale) Mirror 34: 0xe671bdec (stale) Mirror 35: 0xe671bdec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xe671bdec (stale) Mirror 21: 0xe671bdec (stale) Mirror 22: 0xe671bdec (stale) Mirror 23: 0xe671bdec (stale) Mirror 25: 0xe671bdec (stale) Mirror 30: 0xe671bdec (stale) Mirror 31: 0xe671bdec (stale) Mirror 32: 0xe671bdec (stale) Mirror 34: 0xe671bdec (stale) Mirror 35: 0xe671bdec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000908 1788269984.145607 write 0x00d453 thru 0x01006a (0x2c18 bytes) 000913 1788269984.155506 mapwrite 0x032db9 thru 0x03ffff (0xd247 bytes) 000914 1788269984.231816 trunc from 0x040000 to 0x03d65d (0x29a2 bytes) 000915 1788269984.250678 mapwrite 0x011163 thru 0x01784c (0x66ea bytes) 000918 1788269984.307358 write 0x0341c4 thru 0x039d1e (0x5b5b bytes) 000919 1788269984.308163 trunc from 0x03d65e to 0x0239ba (0x19ca3 bytes) 000920 1788269984.324139 MIRROR_RESYNC 11 mirrors 000922 1788269986.275043 read 0x0177db thru 0x021583 (0x9da9 bytes) 000924 1788269986.282381 trunc from 0x0239bb to 0x0187e5 (0xb1d5 bytes) 000927 1788269986.346034 trunc from 0x0187e6 to 0x014231 (0x45b4 bytes) 000928 1788269986.365886 read 0x00b0c1 thru 0x014231 (0x9171 bytes) 000929 1788269986.383714 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14232): Mirror 12: 0xea695ccf (stale) Mirror 21: 0xea695ccf (stale) Mirror 22: 0xea695ccf (stale) Mirror 23: 0xea695ccf (stale) Mirror 25: 0xea695ccf (stale) Mirror 30: 0xea695ccf (stale) Mirror 31: 0xea695ccf (stale) Mirror 34: 0xea695ccf (stale) Mirror 35: 0xea695ccf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14232): Mirror 12: 0xea695ccf (stale) Mirror 21: 0xea695ccf (stale) Mirror 22: 0xea695ccf (stale) Mirror 23: 0xea695ccf (stale) Mirror 25: 0xea695ccf (stale) Mirror 30: 0xea695ccf (stale) Mirror 31: 0xea695ccf (stale) Mirror 34: 0xea695ccf (stale) Mirror 35: 0xea695ccf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14232: skipped 000930 1788269986.798414 read 0x01354d thru 0x014231 (0xce5 bytes) 000933 1788269986.803477 mapwrite 0x00126e thru 0x00cdf6 (0xbb89 bytes) 000934 1788269986.874509 write 0x03d835 thru 0x03ffff (0x27cb bytes) 000936 1788269986.880840 mapwrite 0x0277c4 thru 0x0291b8 (0x19f5 bytes) 000938 1788269986.948634 write 0x01d962 thru 0x02d5c7 (0xfc66 bytes) 000939 1788269986.956427 write 0x03b360 thru 0x03ffff (0x4ca0 bytes) 000941 1788269986.963312 read 0x02ffb3 thru 0x032212 (0x2260 bytes) 000942 1788269986.974051 mapread 0x032197 thru 0x03dbac (0xba16 bytes) 000944 1788269986.975711 trunc from 0x040000 to 0x0258b7 (0x1a748 bytes) 000945 1788269986.994356 trunc from 0x0258b8 to 0x03f5b4 (0x19cfd bytes) 000947 1788269987.019899 read 0x026578 thru 0x02b237 (0x4cc0 bytes) 000948 1788269987.039582 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f5b5): Mirror 21: 0x636c1e2a (stale) Mirror 22: 0x636c1e2a (stale) Mirror 23: 0x636c1e2a (stale) Mirror 25: 0x636c1e2a (stale) Mirror 30: 0x636c1e2a (stale) Mirror 31: 0x636c1e2a (stale) Mirror 34: 0x636c1e2a (stale) Mirror 35: 0x636c1e2a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f5b5): Mirror 21: 0x636c1e2a (stale) Mirror 22: 0x636c1e2a (stale) Mirror 23: 0x636c1e2a (stale) Mirror 25: 0x636c1e2a (stale) Mirror 30: 0x636c1e2a (stale) Mirror 31: 0x636c1e2a (stale) Mirror 34: 0x636c1e2a (stale) Mirror 35: 0x636c1e2a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f5b5: skipped 000949 1788269987.483625 read 0x010a72 thru 0x01c2c1 (0xb850 bytes) 000952 1788269987.501487 mapwrite 0x027138 thru 0x03157a (0xa443 bytes) 000953 1788269987.579537 trunc from 0x03f5b5 to 0x039b10 (0x5aa4 bytes) 000955 1788269987.593461 write 0x0391b6 thru 0x03ffff (0x6e4a bytes) 000956 1788269987.598881 mapread 0x03ffbc thru 0x03ffff (0x44 bytes) 000957 1788269987.600444 mapwrite 0x013c2a thru 0x0221c7 (0xe59e bytes) 000965 1788269987.654043 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x3e1:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x4ffaf57f (stale) Mirror 22: 0x4ffaf57f (stale) Mirror 23: 0x4ffaf57f (stale) Mirror 25: 0x4ffaf57f (stale) Mirror 30: 0x4ffaf57f (stale) Mirror 31: 0x4ffaf57f (stale) Mirror 34: 0x4ffaf57f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x4ffaf57f (stale) Mirror 22: 0x4ffaf57f (stale) Mirror 23: 0x4ffaf57f (stale) Mirror 25: 0x4ffaf57f (stale) Mirror 30: 0x4ffaf57f (stale) Mirror 31: 0x4ffaf57f (stale) Mirror 34: 0x4ffaf57f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000968 1788269988.028310 mapwrite 0x017d35 thru 0x020e4c (0x9118 bytes) 000970 1788269988.099888 MIRROR_RESYNC 8 mirrors 000973 1788269989.133945 mapwrite 0x01b4bd thru 0x029ce6 (0xe82a bytes) 000976 1788269989.232592 trunc from 0x040000 to 0x0237d0 (0x1c82f bytes) 000977 1788269989.254802 mapwrite 0x028988 thru 0x03123f (0x88b8 bytes) 000978 1788269989.307552 read 0x01289e thru 0x014014 (0x1777 bytes) 000980 1788269989.320152 trunc from 0x031240 to 0x03c7c1 (0xb582 bytes) 000982 1788269989.334770 write 0x036a5c thru 0x03ffff (0x95a4 bytes) 000983 1788269989.343317 write 0x02d86e thru 0x03165c (0x3def bytes) 000984 1788269989.344369 mapread 0x01217f thru 0x01aed7 (0x8d59 bytes) 000985 1788269989.345708 write 0x00c8fe thru 0x01c374 (0xfa77 bytes) 000986 1788269989.351421 write 0x019492 thru 0x02568c (0xc1fb bytes) 000987 1788269989.352642 trunc from 0x040000 to 0x00ff01 (0x300fe bytes) 000989 1788269989.363846 mapread 0x0051e9 thru 0x00ed94 (0x9bac bytes) 000994 1788269989.376919 trunc from 0x00ff02 to 0x00621b (0x9ce6 bytes) 000995 1788269989.392249 mapwrite 0x00ab89 thru 0x013830 (0x8ca8 bytes) 000997 1788269989.443264 mapread 0x01107f thru 0x013830 (0x27b2 bytes) 000999 1788269989.444468 mapread 0x0029aa thru 0x00336b (0x9c2 bytes) 001000 1788269989.445262 mapread 0x00314d thru 0x01121c (0xe0d0 bytes) All operations completed A-OK! PASS 70a (86s) == sanity-flr test 71: check mirror extend parent fid ==== 09:39:52 (1788269992) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (1s) == sanity-flr test 72: check append on FLR file ========== 09:39:54 (1788269994) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ea:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109347 s, 19.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5cf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ea:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724501 s, 14.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5cf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5eb:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ea:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5cf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5eb:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x5ea:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5d0:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (5s) == sanity-flr test 200a: stress test ===================== 09:39:58 (1788269998) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3212672 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3793536 .. failed Extending file size to 3518880 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7377696 .. Extending file size to 7767040 .. failed Extending file size to 7476864 .. Extending file size to 8072000 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8379264 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8501568 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7511520 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8747776 .. Extending file size to 8465120 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8801312 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8946592 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9132512 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9264864 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58682 58683 58684 58686 58689 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) 192.168.202.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (69s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:41:07 (1788270067) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6746656 .. Extending file size to 2818912 .. Extending file size to 8345568 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7649024 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8797568 .. Extending file size to 9295648 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9314400 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 60697 60698 60699 60702 60703 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) 192.168.202.110@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (67s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:42:14 (1788270134) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3328 2205056 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3328 2205056 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0446617 s, 47.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103335 s, 20.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa656c375 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:42:22 (1788270142) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0407018 s, 51.5 MB/s PASS 202 (3s) == sanity-flr test 203: mirror file preserve mirror ID === 09:42:25 (1788270145) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x437:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e6:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e6:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e7:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x600:0x0] } PASS 203 (3s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:42:28 (1788270148) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.083676 s, 25.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x601:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x602:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (3s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:42:31 (1788270151) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0572902 s, 36.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ea:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x603:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ea:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x603:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x604:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:42:34 (1788270154) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0687086 s, 30.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x605:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x605:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x606:0x0] } PASS 204c (25s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:42:59 (1788270179) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603414 s, 17.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ec:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x608:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x607:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ec:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x608:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x607:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ed:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:43:46 (1788270226) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x609:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0575966 s, 36.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x609:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x609:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:44:33 (1788270273) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg210-server mds-ost sync done. Creating new pool test_204f oleg210-server: Pool lustre.test_204f created Adding targets to pool oleg210-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg210-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0757258 s, 27.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x60b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg210-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg210-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg210-server: Pool lustre.test_204f destroyed PASS 204f (65s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:45:38 (1788270338) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:45:44 (1788270344) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:45:50 (1788270350) oleg210-server: Pool lustre.test_206 created Adding targets to pool oleg210-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg210-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg210-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg210-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg210-server: Pool lustre.test_206 destroyed PASS 206 (20s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:46:10 (1788270370) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591252 s, 17.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251696 s, 41.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x665b09c5 Mirror 3: 0x665b09c5 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:46:19 (1788270379) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:46:22 (1788270382) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:46:25 (1788270385) Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0510555 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1a50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1400000 Read this (4 bytes): BBBB 192.168.202.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg210-client.virtnet /mnt/lustre2 (opts:) PASS 209a (24s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:46:49 (1788270409) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0157906 s, 519 kB/s the very first read read with warm cache PASS 209b (4s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:46:53 (1788270413) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159442 s, 65.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x45c lmm_fid: [0x200000404:0x45c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x45c lmm_fid: [0x200000404:0x45c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x5f9:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322835036 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-01 09:46:54.000000000 -0400 Modify: 2026-09-01 09:46:54.000000000 -0400 Change: 2026-09-01 09:46:55.000000000 -0400 Birth: 2026-09-01 09:46:54.000000000 -0400 PASS 210a (5s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:46:58 (1788270418) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (2s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:47:00 (1788270420) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.305621 s, 134 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0427177 s, 95.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 1 09:47 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x45e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 1 09:47 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:47:05 (1788270425) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (4s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:47:09 (1788270429) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1083 sec =========== 09:47:15 (1788270435) === sanity-flr: start cleanup 09:47:16 (1788270436) === === sanity-flr: finish cleanup 09:47:20 (1788270440) ===