-----============= acceptance-small: sanity-flr ============----- Mon Aug 31 14:31:24 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg356-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 14:31:41 (1788201101) === oleg356-client.virtnet: executing check_config_client /mnt/lustre oleg356-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg356-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa052052c9000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa052052c9000.idle_timeout=debug disable quota as required oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 14:32:13 (1788201133) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7660] == sanity-flr test 0a: lfs mirror create with -N option == 14:32:16 (1788201136) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:32:38 (1788201158) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:32:41 (1788201161) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:32:44 (1788201164) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:33:10 (1788201190) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:33:13 (1788201193) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 14:33:16 (1788201196) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa052052c9000.stats=clear osc.lustre-OST0001-osc-ffffa052052c9000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:33:24 (1788201204) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:33:36 (1788201216) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:33:45 (1788201225) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:33:48 (1788201228) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:33:55 (1788201235) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 14:34:04 (1788201244) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:34:10 (1788201250) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:34:13 (1788201253) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:34:19 (1788201259) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:34:29 (1788201269) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:34:36 (1788201276) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 2.21925 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904067 s, 11.6 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:34:47 (1788201287) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.17771 s, 29.5 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:34:55 (1788201295) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:35:04 (1788201304) Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa052052c9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa052052c9000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa052052c9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa052052c9000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg356-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (67s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:36:10 (1788201370) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.439337 s, 19.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa052052c9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa052052c9000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:36:55 (1788201415) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa052052c9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa052052c9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa052052c9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa052052c9000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg356-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:38:10 (1788201490) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.67875 s, 11.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 31 14:38 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.222s user 0m0.000s sys 0m0.211s Stopping /mnt/lustre-ost1 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa052052c9000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa052052c9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.087s user 0m0.001s sys 0m0.205s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg356-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg356-server oleg356-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa052052c9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa052052c9000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.505s user 0m0.001s sys 0m0.211s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg356-server: oleg356-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg356-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg356-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa052052c9000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa052052c9000.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:39:42 (1788201582) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:39:46 (1788201586) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:39:49 (1788201589) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 14:39:52 (1788201592) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 14:40:00 (1788201600) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.665734 s, 18.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:40:12 (1788201612) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398929 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251228 s, 41.7 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:40:19 (1788201619) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:40:26 (1788201626) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000837033 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00354539 s, 289 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00128467 s, 797 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0112526 s, 91.0 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 14:40:36 (1788201636) create test files with size 15738011 .. a90b7f6053fcc4aaabe4b37e7718987c - 7444ac53973ecb8732df2b65d12ab362 - 303d458c4eaca29bc38fb6439661cc9a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg356-client.virtnet /mnt/lustre (opts:) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 14:40:59 (1788201659) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:41:16 (1788201676) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:41:22 (1788201682) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989078 s, 10.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200819 s, 52.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 14:41:31 (1788201691) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.089245 s, 23.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.206941 s, 20.3 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.276121 s, 7.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 14:41:48 (1788201708) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 14:41:52 (1788201712) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:41:55 (1788201715) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778074 s, 13.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 14:42:04 (1788201724) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:42:08 (1788201728) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:42:18 (1788201738) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.325217 s, 32.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg356-client.virtnet /mnt/lustre (opts:) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:42:30 (1788201750) llite.lustre-ffffa052071a6000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.307477 s, 34.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg356-client.virtnet /mnt/lustre (opts:) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffffa05210f2a800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:42:43 (1788201763) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:43:19 (1788201799) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:43:56 (1788201836) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 14:44:00 (1788201840) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.253657 s, 12.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.231322 s, 13.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x24f9e884 (nosync,stale) Mirror 2: 0x24f9e884 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6ff79efee4fa60418247470389c8b94c - mirror 1 checksum: 383842ff97cf4397ef6c321f104f3b34 - mirror 2 checksum: 6ff79efee4fa60418247470389c8b94c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-31T14:44:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-31T14:44:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6ff79efee4fa60418247470389c8b94c - mirror 1 checksum: 383842ff97cf4397ef6c321f104f3b34 - mirror 2 checksum: 383842ff97cf4397ef6c321f104f3b34 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-31T14:44:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-31T14:44:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:44:13 (1788201853) Failing mds1 on oleg356-server Stopping /mnt/lustre-mds1 (opts:) on oleg356-server 14:44:21 (1788201861) shut down facet: mds1 facet_host: oleg356-server facet_failover_host: oleg356-server Failover mds1 to oleg356-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Started lustre-MDT0000 14:44:47 (1788201887) targets are mounted 14:44:47 (1788201887) facet_failover done oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249702 s, 42.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (52s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:45:05 (1788201905) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:45:09 (1788201909) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:45:13 (1788201913) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0174112 s, 941 kB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:45:24 (1788201924) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:45:28 (1788201928) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00769785 s, 532 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.183381 s, 22.3 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:45:38 (1788201938) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:46:03 (1788201963) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:46:33 (1788201993) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:46:57 (1788202017) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:47:21 (1788202041) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 14:47:30 (1788202050) mirror create pid 54347 mirror split pid 54348 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54347 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54348 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 14:48:38 (1788202118) Seed set to 2119 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1788202120.091897 mapwrite 0x015ea5 thru 0x0203ff (0xa55b bytes) 000005 1788202120.227624 read 0x008398 thru 0x00975c (0x13c5 bytes) 000006 1788202120.235320 write 0x027e34 thru 0x036601 (0xe7ce bytes) 000008 1788202120.239540 read 0x014708 thru 0x022460 (0xdd59 bytes) truncating to largest ever: 0x32985 000009 1788202120.256594 trunc from 0x036602 to 0x032984 (0x3c7d bytes) 000013 1788202120.285040 mapread 0x01cbfa thru 0x026230 (0x9637 bytes) 000017 1788202120.292417 write 0x007b48 thru 0x0161e4 (0xe69d bytes) 000018 1788202120.301814 MIRROR_EXTEND to 2 mirrors 000019 1788202121.156238 mapwrite 0x02bc05 thru 0x033740 (0x7b3c bytes) 000021 1788202121.434407 write 0x018765 thru 0x0218a1 (0x913d bytes) 000022 1788202121.451657 read 0x0088a3 thru 0x00a5e9 (0x1d47 bytes) 000023 1788202121.464641 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33741): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33741): Mirror 2: 0x6d308d4c (stale) CRC-32 checksum value for chunk [0, 0x33741): Mirror 1: 0x2fec236c Mirror 3: 0x2fec236c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33741: skipped 000024 1788202122.416492 read 0x0251b4 thru 0x02a839 (0x5686 bytes) 000025 1788202122.433078 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x33741): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33741): Mirror 2: 0x6d308d4c (stale) CRC-32 checksum value for chunk [0, 0x33741): Mirror 1: 0x2fec236c Mirror 3: 0x2fec236c Mirror 4: 0x2fec236c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33741: skipped truncating to largest ever: 0x33d9c 000026 1788202123.347509 trunc from 0x033741 to 0x033d9b (0x65b bytes) 000027 1788202123.474465 mapread 0x024364 thru 0x032f66 (0xec03 bytes) 000030 1788202123.521200 read 0x0141aa thru 0x020c44 (0xca9b bytes) 000031 1788202123.541110 write 0x00f3a0 thru 0x0171eb (0x7e4c bytes) 000032 1788202123.549736 mapwrite 0x026294 thru 0x02db95 (0x7902 bytes) 000035 1788202123.607655 mapwrite 0x01db19 thru 0x02c1c6 (0xe6ae bytes) 000036 1788202123.657634 mapread 0x00ddef thru 0x01112e (0x3340 bytes) 000038 1788202123.672624 write 0x03ecc4 thru 0x03ffff (0x133c bytes) 000039 1788202123.675206 write 0x03d72c thru 0x03ffff (0x28d4 bytes) 000041 1788202123.683654 write 0x02fdb1 thru 0x038657 (0x88a7 bytes) 000042 1788202123.691090 read 0x0360a9 thru 0x03d591 (0x74e9 bytes) 000043 1788202123.699502 write 0x03853e thru 0x03ffff (0x7ac2 bytes) 000045 1788202123.702587 write 0x0228c6 thru 0x02639b (0x3ad6 bytes) 000046 1788202123.704218 read 0x0132c2 thru 0x017801 (0x4540 bytes) truncating to largest ever: 0x39d7b 000047 1788202123.705338 trunc from 0x040000 to 0x039d7a (0x6285 bytes) 000048 1788202123.733001 trunc from 0x039d7b to 0x001031 (0x38d49 bytes) 000049 1788202123.758192 read 0x0000a7 thru 0x001031 (0xf8b bytes) 000051 1788202123.760447 mapwrite 0x00474c thru 0x007ae6 (0x339b bytes) 000052 1788202123.873659 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x7ae7): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7ae7): Mirror 2: 0x64e8ac6d (stale) Mirror 3: 0x64e8ac6d (stale) Mirror 4: 0x64e8ac6d (stale) CRC-32 checksum value for chunk [0, 0x7ae7): Mirror 1: 0xdcff51e8 Mirror 5: 0xdcff51e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7ae7: skipped 000054 1788202125.147316 trunc from 0x007ae7 to 0x013df4 (0xc30e bytes) 000055 1788202125.262498 trunc from 0x013df5 to 0x00fa38 (0x43bc bytes) 000056 1788202125.302199 trunc from 0x00fa39 to 0x02bd8b (0x1c353 bytes) 000060 1788202125.324546 mapwrite 0x031c68 thru 0x03ea40 (0xcdd9 bytes) 000061 1788202125.506042 trunc from 0x03ea41 to 0x014891 (0x2a1af bytes) 000062 1788202125.544548 read 0x003557 thru 0x003ec2 (0x96c bytes) 000063 1788202125.553143 mapread 0x00f3a3 thru 0x014891 (0x54ef bytes) 000065 1788202125.568191 write 0x0184dd thru 0x0211a7 (0x8ccb bytes) 000066 1788202125.570492 read 0x00d9e7 thru 0x014e89 (0x74a3 bytes) 000068 1788202125.572050 mapwrite 0x009b5d thru 0x00ac5d (0x1101 bytes) 000069 1788202125.650967 write 0x02f782 thru 0x03a346 (0xabc5 bytes) 000070 1788202125.655255 read 0x00421c thru 0x00e767 (0xa54c bytes) 000071 1788202125.673135 read 0x039982 thru 0x03a346 (0x9c5 bytes) 000072 1788202125.673910 mapwrite 0x03c5d1 thru 0x03ffff (0x3a2f bytes) 000075 1788202125.796373 mapwrite 0x0259eb thru 0x02ef60 (0x9576 bytes) 000077 1788202125.915070 write 0x0341ed thru 0x03f75a (0xb56e bytes) 000078 1788202125.922209 trunc from 0x040000 to 0x00dd2c (0x322d3 bytes) 000079 1788202125.964579 trunc from 0x00dd2d to 0x017037 (0x930b bytes) 000080 1788202125.990311 mapread 0x000880 thru 0x008bab (0x832c bytes) 000082 1788202125.992295 mapread 0x013e27 thru 0x016b3e (0x2d18 bytes) 000083 1788202126.044239 write 0x01f626 thru 0x023f5e (0x4939 bytes) 000086 1788202126.060201 trunc from 0x023f5f to 0x002866 (0x216f8 bytes) 000088 1788202126.099296 MIRROR_RESYNC 5 mirrors 000089 1788202127.466648 MIRROR_RESYNC 5 mirrors 000092 1788202127.931642 read 0x000e1a thru 0x002866 (0x1a4d bytes) 000093 1788202127.943706 mapread 0x000ea2 thru 0x002866 (0x19c5 bytes) 000095 1788202127.948307 read 0x00196e thru 0x002866 (0xef9 bytes) 000098 1788202127.956054 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000099 1788202128.481243 mapread 0x00136e thru 0x002866 (0x14f9 bytes) 000101 1788202128.501912 trunc from 0x002867 to 0x02d1bd (0x2a957 bytes) 000102 1788202128.603405 MIRROR_RESYNC 4 mirrors 000105 1788202129.846642 mapread 0x028249 thru 0x02d1bd (0x4f75 bytes) 000109 1788202129.887045 write 0x03196f thru 0x039cf9 (0x838b bytes) 000111 1788202129.974593 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39cfa): Mirror 2: 0x307d902 (stale) Mirror 4: 0x307d902 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39cfa): Mirror 2: 0x307d902 (stale) Mirror 4: 0x307d902 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39cfa: skipped 000113 1788202130.554540 mapread 0x00c322 thru 0x01b0ab (0xed8a bytes) 000114 1788202130.587267 trunc from 0x039cfa to 0x00b5f7 (0x2e702 bytes) 000115 1788202130.604672 read 0x00ad39 thru 0x00b5f7 (0x8bf bytes) 000119 1788202130.608653 read 0x006be3 thru 0x00b5f7 (0x4a15 bytes) 000120 1788202130.618193 mapwrite 0x02367d thru 0x027215 (0x3b99 bytes) 000122 1788202130.766836 trunc from 0x027216 to 0x038cb7 (0x11aa2 bytes) 000123 1788202130.802142 MIRROR_RESYNC 3 mirrors 000124 1788202132.267245 write 0x00e6e7 thru 0x01d432 (0xed4c bytes) 000125 1788202132.355510 read 0x005353 thru 0x006bbc (0x186a bytes) 000128 1788202132.368018 mapwrite 0x024fd3 thru 0x02b093 (0x60c1 bytes) truncating to largest ever: 0x3ba01 000130 1788202132.451517 trunc from 0x038cb8 to 0x03ba00 (0x2d49 bytes) 000131 1788202132.480871 mapread 0x032dba thru 0x03a626 (0x786d bytes) 000132 1788202132.505146 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ba01): Mirror 2: 0x9f41a726 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ba01): Mirror 2: 0x9f41a726 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ba01: skipped 000133 1788202132.913439 write 0x025118 thru 0x031b76 (0xca5f bytes) 000135 1788202132.938520 write 0x03134e thru 0x03ffff (0xecb2 bytes) 000136 1788202132.941372 trunc from 0x040000 to 0x023f63 (0x1c09c bytes) 000137 1788202132.973601 read 0x020d5b thru 0x023f63 (0x3209 bytes) 000139 1788202132.996461 trunc from 0x023f64 to 0x0374de (0x1357b bytes) 000140 1788202133.026119 read 0x0147f0 thru 0x0240e8 (0xf8f9 bytes) 000145 1788202133.077776 trunc from 0x0374df to 0x037da5 (0x8c7 bytes) 000147 1788202133.110109 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000150 1788202133.239397 mapread 0x0022c9 thru 0x009607 (0x733f bytes) 000153 1788202133.261828 mapwrite 0x00e919 thru 0x0195e6 (0xacce bytes) 000154 1788202133.324428 mapread 0x0364e6 thru 0x037da5 (0x18c0 bytes) 000156 1788202133.334445 write 0x037c5a thru 0x03ffff (0x83a6 bytes) 000159 1788202133.338798 mapwrite 0x010935 thru 0x018438 (0x7b04 bytes) 000160 1788202133.382775 mapread 0x03c841 thru 0x03ffff (0x37bf bytes) 000161 1788202133.384186 mapread 0x01bb12 thru 0x02a016 (0xe505 bytes) 000162 1788202133.429939 write 0x03517a thru 0x0355de (0x465 bytes) 000165 1788202133.433401 mapread 0x030b4b thru 0x034c35 (0x40eb bytes) 000166 1788202133.434653 mapwrite 0x03845e thru 0x03ffff (0x7ba2 bytes) 000167 1788202133.526927 mapread 0x03a022 thru 0x03e38b (0x436a bytes) 000170 1788202133.530798 mapwrite 0x036887 thru 0x03ffff (0x9779 bytes) 000174 1788202133.637937 read 0x00cf94 thru 0x01b855 (0xe8c2 bytes) 000175 1788202133.640525 trunc from 0x040000 to 0x01bd24 (0x242db bytes) 000177 1788202133.683209 write 0x036fdb thru 0x03ffff (0x9025 bytes) 000178 1788202133.686425 mapread 0x0194b7 thru 0x01e900 (0x544a bytes) 000179 1788202133.697988 mapread 0x00f15c thru 0x01392d (0x47d2 bytes) 000180 1788202133.699344 mapwrite 0x0323b7 thru 0x03c3c0 (0xa00a bytes) 000183 1788202133.796032 mapread 0x0056bb thru 0x00d2c0 (0x7c06 bytes) 000184 1788202133.808018 trunc from 0x040000 to 0x0209b8 (0x1f647 bytes) 000186 1788202133.835522 read 0x0113b3 thru 0x0159fe (0x464c bytes) 000187 1788202133.847046 MIRROR_EXTEND to 2 mirrors 000190 1788202134.949070 mapwrite 0x00b4fb thru 0x00b6fb (0x201 bytes) 000191 1788202135.121757 write 0x03c7bc thru 0x03ffff (0x3844 bytes) 000192 1788202135.124484 read 0x003c08 thru 0x005a5a (0x1e53 bytes) 000194 1788202135.132122 mapread 0x00c9fc thru 0x01b259 (0xe85e bytes) 000195 1788202135.151398 mapread 0x003ec7 thru 0x0101ed (0xc327 bytes) 000196 1788202135.153171 write 0x02230c thru 0x0251fb (0x2ef0 bytes) 000197 1788202135.175403 write 0x02db7c thru 0x0310ac (0x3531 bytes) 000198 1788202135.187942 mapread 0x03ae74 thru 0x03c9d5 (0x1b62 bytes) truncating to largest ever: 0x3f802 000199 1788202135.202420 trunc from 0x040000 to 0x03f801 (0x7fe bytes) 000201 1788202135.236987 read 0x00f773 thru 0x00fa5d (0x2eb bytes) 000202 1788202135.240616 mapwrite 0x02c85e thru 0x034ee4 (0x8687 bytes) 000203 1788202135.320186 trunc from 0x03f802 to 0x02b8f6 (0x13f0b bytes) 000207 1788202135.351472 write 0x03bc3e thru 0x03fd4f (0x4112 bytes) 000208 1788202135.362398 mapwrite 0x015625 thru 0x019035 (0x3a11 bytes) 000211 1788202135.417390 read 0x001051 thru 0x00a28a (0x923a bytes) 000212 1788202135.421053 mapread 0x013e44 thru 0x016329 (0x24e6 bytes) 000213 1788202135.423181 mapwrite 0x00d971 thru 0x01aec5 (0xd555 bytes) 000214 1788202135.505689 mapwrite 0x034aef thru 0x03ffff (0xb511 bytes) 000215 1788202135.615069 mapread 0x030d0e thru 0x033cbb (0x2fae bytes) 000217 1788202135.619434 read 0x022d37 thru 0x02ffad (0xd277 bytes) 000218 1788202135.620972 trunc from 0x040000 to 0x03f3ba (0xc45 bytes) 000220 1788202135.648650 trunc from 0x03f3bb to 0x013935 (0x2ba85 bytes) 000222 1788202135.666682 write 0x0360b1 thru 0x03ffff (0x9f4f bytes) 000223 1788202135.669606 mapwrite 0x021858 thru 0x031430 (0xfbd9 bytes) 000225 1788202135.737740 mapread 0x007695 thru 0x00bde2 (0x474e bytes) 000227 1788202135.742772 mapwrite 0x0164c2 thru 0x024939 (0xe478 bytes) 000229 1788202135.798075 trunc from 0x040000 to 0x0151e6 (0x2ae19 bytes) 000230 1788202135.815583 trunc from 0x0151e7 to 0x015ae2 (0x8fc bytes) 000231 1788202135.837640 trunc from 0x015ae3 to 0x03a6c2 (0x24be0 bytes) 000233 1788202135.858444 read 0x0292dd thru 0x034a7a (0xb79e bytes) 000234 1788202135.868595 trunc from 0x03a6c3 to 0x037104 (0x35be bytes) 000237 1788202135.898825 mapread 0x01ca1f thru 0x0209d5 (0x3fb7 bytes) 000238 1788202135.917785 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000239 1788202136.159375 mapwrite 0x00e680 thru 0x01ad54 (0xc6d5 bytes) 000240 1788202136.318321 mapread 0x02b2e5 thru 0x037104 (0xbe20 bytes) 000241 1788202136.354785 read 0x01f324 thru 0x01fb10 (0x7ed bytes) 000242 1788202136.361830 read 0x009b1e thru 0x0197df (0xfcc2 bytes) 000244 1788202136.374572 mapwrite 0x00e3e5 thru 0x01833d (0x9f59 bytes) 000245 1788202136.475718 mapread 0x020c03 thru 0x02fb31 (0xef2f bytes) 000247 1788202136.502952 mapwrite 0x02aa76 thru 0x02bbc4 (0x114f bytes) 000248 1788202136.562520 write 0x01b437 thru 0x02a697 (0xf261 bytes) 000249 1788202136.564780 write 0x01955c thru 0x025adb (0xc580 bytes) 000250 1788202136.569597 mapread 0x02b069 thru 0x034707 (0x969f bytes) 000251 1788202136.571961 mapread 0x01a78b thru 0x02847b (0xdcf1 bytes) 000254 1788202136.576549 mapread 0x00e6cc thru 0x01a526 (0xbe5b bytes) 000255 1788202136.577958 mapwrite 0x033998 thru 0x0360ea (0x2753 bytes) 000256 1788202136.647137 read 0x00aab2 thru 0x00b907 (0xe56 bytes) 000257 1788202136.658772 MIRROR_EXTEND to 2 mirrors 000260 1788202137.820356 read 0x00ab87 thru 0x00b11b (0x595 bytes) 000262 1788202137.878530 trunc from 0x037105 to 0x01f1b9 (0x17f4b bytes) 000265 1788202138.028246 write 0x00b9b6 thru 0x01446a (0x8ab5 bytes) 000266 1788202138.050516 write 0x03a3dc thru 0x03ffff (0x5c24 bytes) 000267 1788202138.052611 read 0x004adf thru 0x010349 (0xb86b bytes) 000268 1788202138.083416 read 0x03fe63 thru 0x03ffff (0x19d bytes) 000270 1788202138.091014 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3042d5e0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb553ab4b Mirror 3: 0xb553ab4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000271 1788202139.899726 read 0x03389f thru 0x03658e (0x2cf0 bytes) 000272 1788202139.910997 mapread 0x021de3 thru 0x02a53c (0x875a bytes) 000273 1788202139.939589 trunc from 0x040000 to 0x03aae9 (0x5516 bytes) 000274 1788202140.068276 MIRROR_RESYNC 3 mirrors 000280 1788202141.127179 mapread 0x0235f3 thru 0x02a564 (0x6f72 bytes) 000282 1788202141.152730 mapread 0x02397e thru 0x027c2e (0x42b1 bytes) 000283 1788202141.153581 read 0x02e4d2 thru 0x035c1b (0x774a bytes) 000284 1788202141.176986 write 0x03f013 thru 0x03ffff (0xfed bytes) 000286 1788202141.269615 mapwrite 0x00455d thru 0x00ce9b (0x893f bytes) 000287 1788202141.387045 mapread 0x00d637 thru 0x013884 (0x624e bytes) 000293 1788202141.421402 write 0x000eee thru 0x00810c (0x721f bytes) 000294 1788202141.438445 mapread 0x02980e thru 0x031d39 (0x852c bytes) 000295 1788202141.465512 mapread 0x00a34b thru 0x00aa77 (0x72d bytes) 000297 1788202141.477526 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9f1dd97 (stale) Mirror 3: 0x9f1dd97 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd4a47fc9 Mirror 4: 0xd4a47fc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000299 1788202142.907942 write 0x021967 thru 0x029369 (0x7a03 bytes) 000300 1788202143.090105 mapread 0x029d02 thru 0x03664d (0xc94c bytes) 000301 1788202143.194348 trunc from 0x040000 to 0x00a2d2 (0x35d2d bytes) 000303 1788202143.232178 mapread 0x0015bd thru 0x003e04 (0x2848 bytes) 000304 1788202143.259698 write 0x0233dd thru 0x024afa (0x171e bytes) 000306 1788202143.261428 write 0x03dd02 thru 0x03ffff (0x22fe bytes) 000307 1788202143.285936 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9f1dd97 (stale) Mirror 3: 0x9f1dd97 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9f1dd97 (stale) Mirror 3: 0x9f1dd97 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000308 1788202143.640458 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9f1dd97 (stale) Mirror 3: 0x9f1dd97 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8cbeb291 Mirror 4: 0x8cbeb291 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000309 1788202144.982013 mapwrite 0x033ccb thru 0x03c735 (0x8a6b bytes) 000311 1788202145.279211 write 0x036e97 thru 0x03ffff (0x9169 bytes) 000314 1788202145.283827 mapwrite 0x025e17 thru 0x02f626 (0x9810 bytes) 000315 1788202145.382571 mapread 0x033281 thru 0x03f887 (0xc607 bytes) 000317 1788202145.395759 MIRROR_RESYNC 4 mirrors 000318 1788202146.836557 mapread 0x019ff7 thru 0x025285 (0xb28f bytes) 000323 1788202146.871532 read 0x00a8ef thru 0x00f3ca (0x4adc bytes) 000324 1788202146.887979 write 0x00ff3e thru 0x01558b (0x564e bytes) 000326 1788202147.004546 trunc from 0x040000 to 0x007a2b (0x385d4 bytes) 000327 1788202147.053418 mapread 0x005c88 thru 0x007a2b (0x1da4 bytes) 000328 1788202147.069692 write 0x03f37d thru 0x03ffff (0xc83 bytes) 000330 1788202147.073620 write 0x022d50 thru 0x02874e (0x59ff bytes) 000335 1788202147.108069 mapwrite 0x012bc3 thru 0x01446a (0x18a8 bytes) 000337 1788202147.247144 write 0x018d0c thru 0x027e6e (0xf163 bytes) 000338 1788202147.257329 MIRROR_RESYNC 4 mirrors 000340 1788202148.280620 read 0x037d53 thru 0x03ffff (0x82ad bytes) 000341 1788202148.308457 write 0x02e4ef thru 0x0372e5 (0x8df7 bytes) 000347 1788202148.459393 write 0x01599a thru 0x01e7eb (0x8e52 bytes) 000350 1788202148.492094 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6593dc51 (stale) Mirror 3: 0x6593dc51 (stale) Mirror 4: 0x6593dc51 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdc36b334 Mirror 5: 0xdc36b334 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000354 1788202149.722053 write 0x0166c3 thru 0x0252e3 (0xec21 bytes) 000355 1788202149.856826 mapwrite 0x001f36 thru 0x0102ef (0xe3ba bytes) 000357 1788202149.945298 mapwrite 0x02cbf2 thru 0x02d0b4 (0x4c3 bytes) 000358 1788202150.004026 mapread 0x034b27 thru 0x03615c (0x1636 bytes) 000359 1788202150.023317 mapread 0x0195fd thru 0x020054 (0x6a58 bytes) 000360 1788202150.025645 trunc from 0x040000 to 0x011e1d (0x2e1e2 bytes) 000361 1788202150.055531 trunc from 0x011e1e to 0x034631 (0x22814 bytes) 000363 1788202150.092356 trunc from 0x034632 to 0x017871 (0x1cdc0 bytes) 000364 1788202150.115769 trunc from 0x017872 to 0x038f69 (0x216f8 bytes) 000369 1788202150.187154 write 0x030b21 thru 0x032e34 (0x2314 bytes) 000371 1788202150.208805 trunc from 0x038f6a to 0x00ebe2 (0x2a387 bytes) 000373 1788202150.247945 MIRROR_RESYNC 5 mirrors 000374 1788202151.312012 mapread 0x00385c thru 0x00ebe2 (0xb387 bytes) 000375 1788202151.347047 write 0x03c6fd thru 0x03ffff (0x3903 bytes) 000376 1788202151.432613 mapread 0x0263d9 thru 0x02e7ce (0x83f6 bytes) 000377 1788202151.446279 read 0x03650e thru 0x03ca86 (0x6579 bytes) 000379 1788202151.457373 read 0x003ef2 thru 0x00f170 (0xb27f bytes) 000384 1788202151.474025 mapread 0x03d9b8 thru 0x03ffff (0x2648 bytes) 000386 1788202151.475715 mapwrite 0x03f64e thru 0x03ffff (0x9b2 bytes) 000388 1788202151.538133 trunc from 0x040000 to 0x00e3a8 (0x31c57 bytes) 000389 1788202151.576602 read 0x004956 thru 0x00e3a8 (0x9a53 bytes) 000390 1788202151.578357 trunc from 0x00e3a9 to 0x03d182 (0x2edda bytes) 000391 1788202151.633504 mapread 0x001446 thru 0x0094fa (0x80b5 bytes) 000392 1788202151.640967 mapwrite 0x03838a thru 0x03ffff (0x7c76 bytes) 000394 1788202151.838024 trunc from 0x040000 to 0x003a22 (0x3c5dd bytes) 000395 1788202151.871690 trunc from 0x003a23 to 0x00f2bc (0xb89a bytes) 000396 1788202151.890013 read 0x007ac7 thru 0x008dcd (0x1307 bytes) 000397 1788202151.906204 mapwrite 0x014cd3 thru 0x016c13 (0x1f41 bytes) 000400 1788202152.058682 mapwrite 0x0316d6 thru 0x0338c2 (0x21ed bytes) 000401 1788202152.170391 mapwrite 0x034bee thru 0x03ffff (0xb412 bytes) 000402 1788202152.265588 mapread 0x00ff94 thru 0x019517 (0x9584 bytes) 000404 1788202152.270306 write 0x01d432 thru 0x0288d6 (0xb4a5 bytes) 000405 1788202152.271787 write 0x00f15f thru 0x01c05d (0xceff bytes) 000406 1788202152.274702 write 0x01b3b8 thru 0x01c882 (0x14cb bytes) 000408 1788202152.282153 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8aee2f65 (stale) Mirror 4: 0x8aee2f65 (stale) Mirror 5: 0x8aee2f65 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8aee2f65 (stale) Mirror 4: 0x8aee2f65 (stale) Mirror 5: 0x8aee2f65 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000409 1788202152.862657 trunc from 0x040000 to 0x034623 (0xb9dc bytes) 000413 1788202152.905501 read 0x007dc7 thru 0x01136e (0x95a8 bytes) 000414 1788202152.923724 trunc from 0x034624 to 0x00bcd9 (0x2894a bytes) 000417 1788202152.961384 write 0x00adf7 thru 0x00c65b (0x1865 bytes) 000419 1788202152.976722 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc65c): Mirror 4: 0x4c4b4c32 (stale) Mirror 5: 0x4c4b4c32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc65c): Mirror 4: 0x4c4b4c32 (stale) Mirror 5: 0x4c4b4c32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc65c: skipped 000420 1788202153.471778 read 0x0047ec thru 0x00c65b (0x7e70 bytes) 000421 1788202153.497731 mapread 0x008794 thru 0x00c65b (0x3ec8 bytes) 000422 1788202153.515547 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xc65c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc65c): Mirror 4: 0x4c4b4c32 (stale) Mirror 5: 0x4c4b4c32 (stale) CRC-32 checksum value for chunk [0, 0xc65c): Mirror 1: 0x2ba19861 Mirror 6: 0x2ba19861 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc65c: skipped 000423 1788202154.833748 mapwrite 0x00e0f1 thru 0x014016 (0x5f26 bytes) 000425 1788202155.034022 write 0x03c264 thru 0x03ffff (0x3d9c bytes) 000426 1788202155.037949 read 0x02f8e5 thru 0x03845a (0x8b76 bytes) 000427 1788202155.047933 mapread 0x01387b thru 0x01887f (0x5005 bytes) 000428 1788202155.057257 read 0x01438c thru 0x021642 (0xd2b7 bytes) 000429 1788202155.074004 mapwrite 0x0125eb thru 0x021fa2 (0xf9b8 bytes) 000430 1788202155.171897 MIRROR_RESYNC 4 mirrors 000431 1788202156.451270 mapwrite 0x03b6be thru 0x03ffff (0x4942 bytes) 000432 1788202156.733057 read 0x01bb6f thru 0x02906b (0xd4fd bytes) 000436 1788202156.754760 mapwrite 0x016918 thru 0x023fc4 (0xd6ad bytes) 000439 1788202156.844125 trunc from 0x040000 to 0x03914e (0x6eb1 bytes) 000440 1788202156.881393 MIRROR_RESYNC 4 mirrors 000441 1788202157.785459 trunc from 0x03914f to 0x00d199 (0x2bfb5 bytes) 000443 1788202157.871511 trunc from 0x00d19a to 0x03242b (0x25292 bytes) 000444 1788202157.901364 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3242c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3242c): Mirror 4: 0x774b06af (stale) Mirror 5: 0x774b06af (stale) Mirror 6: 0x774b06af (stale) CRC-32 checksum value for chunk [0, 0x3242c): Mirror 1: 0xbdc7f0d1 Mirror 7: 0xbdc7f0d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3242c: skipped 000445 1788202158.973560 mapwrite 0x02e9ec thru 0x03751b (0x8b30 bytes) 000446 1788202159.251227 mapwrite 0x01bf91 thru 0x023044 (0x70b4 bytes) 000447 1788202159.349784 trunc from 0x03751c to 0x004cae (0x3286d bytes) 000448 1788202159.405104 mapread 0x003844 thru 0x004cae (0x146b bytes) 000449 1788202159.423485 mapread 0x0020f8 thru 0x004cae (0x2bb7 bytes) 000450 1788202159.429717 read 0x004b52 thru 0x004cae (0x15d bytes) 000451 1788202159.446639 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4caf): Mirror 4: 0x25f96516 (stale) Mirror 5: 0x25f96516 (stale) Mirror 6: 0x25f96516 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4caf): Mirror 4: 0x25f96516 (stale) Mirror 5: 0x25f96516 (stale) Mirror 6: 0x25f96516 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4caf: skipped 000452 1788202159.948708 mapwrite 0x038c5a thru 0x03ffff (0x73a6 bytes) 000453 1788202160.114464 trunc from 0x040000 to 0x034aa4 (0xb55b bytes) 000454 1788202160.166394 mapread 0x02c525 thru 0x034aa4 (0x8580 bytes) 000455 1788202160.186692 mapwrite 0x0353f0 thru 0x03ffff (0xac10 bytes) 000456 1788202160.347997 mapwrite 0x02beb5 thru 0x038d23 (0xce6f bytes) 000457 1788202160.434113 write 0x028bc4 thru 0x0370f7 (0xe534 bytes) 000459 1788202160.451418 trunc from 0x040000 to 0x00c25d (0x33da2 bytes) 000460 1788202160.508019 mapwrite 0x03deaf thru 0x03ffff (0x2151 bytes) 000461 1788202160.708353 mapwrite 0x029c9c thru 0x036cfe (0xd063 bytes) 000462 1788202160.779587 mapread 0x034cb6 thru 0x03b059 (0x63a4 bytes) 000465 1788202160.781876 mapwrite 0x023692 thru 0x031b95 (0xe504 bytes) 000466 1788202160.837068 mapwrite 0x01630d thru 0x01cb70 (0x6864 bytes) 000467 1788202160.969219 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x970f70a (stale) Mirror 5: 0x970f70a (stale) Mirror 6: 0x970f70a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2485042a Mirror 7: 0x2485042a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000468 1788202162.134561 read 0x030dca thru 0x03a6fe (0x9935 bytes) 000470 1788202162.162698 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x970f70a (stale) Mirror 6: 0x970f70a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x970f70a (stale) Mirror 6: 0x970f70a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2485042a Mirror 7: 0x2485042a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000471 1788202162.958721 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x970f70a (stale) Mirror 6: 0x970f70a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x970f70a (stale) Mirror 6: 0x970f70a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000472 1788202163.312222 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x970f70a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x970f70a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000473 1788202163.701790 read 0x028133 thru 0x0324a0 (0xa36e bytes) 000474 1788202163.715531 read 0x033d55 thru 0x03ebd5 (0xae81 bytes) 000475 1788202163.745415 trunc from 0x040000 to 0x0056ae (0x3a951 bytes) 000477 1788202163.797266 mapread 0x004a9e thru 0x0056ae (0xc11 bytes) 000479 1788202163.817828 write 0x01c548 thru 0x01e2e7 (0x1da0 bytes) 000480 1788202163.819118 mapread 0x00ea17 thru 0x0108e0 (0x1eca bytes) 000481 1788202163.848831 mapread 0x005ef1 thru 0x012e9d (0xcfad bytes) 000483 1788202163.993167 write 0x00d756 thru 0x01aa07 (0xd2b2 bytes) 000484 1788202164.004231 trunc from 0x01e2e8 to 0x0070bf (0x17228 bytes) 000487 1788202164.026685 mapwrite 0x0204a7 thru 0x024f4a (0x4aa4 bytes) 000488 1788202164.128029 read 0x00b051 thru 0x01a25b (0xf20b bytes) 000490 1788202164.166917 mapread 0x002740 thru 0x00743e (0x4cff bytes) 000492 1788202164.174698 MIRROR_RESYNC 2 mirrors 000493 1788202165.269937 MIRROR_EXTEND to 3 mirrors 000496 1788202166.289940 write 0x025d35 thru 0x02718a (0x1456 bytes) 000497 1788202166.375945 mapread 0x01926b thru 0x0226a4 (0x943a bytes) 000501 1788202166.407780 mapread 0x006347 thru 0x008cf2 (0x29ac bytes) 000503 1788202166.418313 mapwrite 0x03f87b thru 0x03ffff (0x785 bytes) 000504 1788202166.548424 write 0x018bcf thru 0x01b154 (0x2586 bytes) 000505 1788202166.555561 trunc from 0x040000 to 0x026349 (0x19cb6 bytes) 000506 1788202166.583829 mapwrite 0x020722 thru 0x02c01c (0xb8fb bytes) 000507 1788202166.764807 write 0x03afee thru 0x03cc83 (0x1c96 bytes) 000508 1788202166.769776 mapwrite 0x025368 thru 0x02bcef (0x6988 bytes) 000509 1788202166.871577 write 0x023694 thru 0x028b1e (0x548b bytes) 000512 1788202166.879655 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3cc84): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc84): Mirror 6: 0xca4b850 (stale) Mirror 7: 0xca4b850 (stale) CRC-32 checksum value for chunk [0, 0x3cc84): Mirror 1: 0x2fad8add Mirror 8: 0x2fad8add lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc84: skipped 000515 1788202167.928317 write 0x016462 thru 0x024b38 (0xe6d7 bytes) 000518 1788202168.018688 mapwrite 0x00851c thru 0x010052 (0x7b37 bytes) 000520 1788202168.188988 trunc from 0x03cc84 to 0x0107fb (0x2c488 bytes) 000522 1788202168.223280 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x107fc): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x107fc): Mirror 6: 0x7415fc18 (stale) Mirror 7: 0x7415fc18 (stale) Mirror 8: 0x7415fc18 (stale) CRC-32 checksum value for chunk [0, 0x107fc): Mirror 1: 0x5a205ad8 Mirror 9: 0x5a205ad8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x107fc: skipped 000526 1788202169.661513 read 0x006de6 thru 0x00dd97 (0x6fb2 bytes) 000527 1788202169.704985 trunc from 0x0107fc to 0x000365 (0x10496 bytes) 000528 1788202169.864320 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x366): Mirror 7: 0x22fa71a7 (stale) Mirror 8: 0x22fa71a7 (stale) Mirror 9: 0x22fa71a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366): Mirror 7: 0x22fa71a7 (stale) Mirror 8: 0x22fa71a7 (stale) Mirror 9: 0x22fa71a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366: skipped 000529 1788202170.417438 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x366): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366): Mirror 7: 0x22fa71a7 (stale) Mirror 8: 0x22fa71a7 (stale) Mirror 9: 0x22fa71a7 (stale) CRC-32 checksum value for chunk [0, 0x366): Mirror 1: 0x22fa71a7 Mirror 10: 0x22fa71a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366: skipped 000530 1788202171.841330 trunc from 0x000366 to 0x011557 (0x111f2 bytes) 000531 1788202171.986340 mapwrite 0x01b2ec thru 0x020ae4 (0x57f9 bytes) 000532 1788202172.175941 mapread 0x019d24 thru 0x020ae4 (0x6dc1 bytes) 000533 1788202172.177742 trunc from 0x020ae5 to 0x01fef5 (0xbef bytes) 000534 1788202172.209774 read 0x00b147 thru 0x018277 (0xd131 bytes) 000535 1788202172.227675 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1fef6): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fef6): Mirror 7: 0x2324c33 (stale) Mirror 8: 0x388987ca (stale) Mirror 9: 0x24695e79 (stale) Mirror 10: 0xc4866cc (stale) CRC-32 checksum value for chunk [0, 0x1fef6): Mirror 1: 0x30151141 Mirror 11: 0x30151141 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fef6: skipped 000536 1788202173.424011 trunc from 0x01fef6 to 0x0293d7 (0x94e2 bytes) 000538 1788202173.589402 write 0x018967 thru 0x01a2e3 (0x197d bytes) 000539 1788202173.603835 mapwrite 0x000f9b thru 0x002662 (0x16c8 bytes) 000540 1788202173.718127 mapread 0x01ab21 thru 0x0293d7 (0xe8b7 bytes) 000542 1788202173.749981 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x293d8): Mirror 8: 0xd04ece60 (stale) Mirror 9: 0xa72aabec (stale) Mirror 10: 0x9e600c89 (stale) Mirror 11: 0x5d8a6c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x293d8): Mirror 8: 0xd04ece60 (stale) Mirror 9: 0xa72aabec (stale) Mirror 10: 0x9e600c89 (stale) Mirror 11: 0x5d8a6c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x293d8: skipped 000544 1788202174.196758 read 0x0036bd thru 0x01103b (0xd97f bytes) 000546 1788202174.222571 write 0x00745f thru 0x008b5c (0x16fe bytes) 000548 1788202174.228881 trunc from 0x0293d8 to 0x03cad8 (0x13701 bytes) 000550 1788202174.256471 mapread 0x0029f6 thru 0x00e59d (0xbba8 bytes) 000552 1788202174.276879 read 0x0250c3 thru 0x025d85 (0xcc3 bytes) 000555 1788202174.316575 mapread 0x01d47d thru 0x01e0fb (0xc7f bytes) 000556 1788202174.317680 write 0x0346f3 thru 0x03ff54 (0xb862 bytes) 000557 1788202174.319068 read 0x00a95a thru 0x015dff (0xb4a6 bytes) 000558 1788202174.322395 mapread 0x033c41 thru 0x03d938 (0x9cf8 bytes) 000559 1788202174.324648 mapread 0x00bb68 thru 0x011bbf (0x6058 bytes) 000560 1788202174.331114 MIRROR_RESYNC 5 mirrors 000561 1788202175.996861 mapread 0x00274b thru 0x00de16 (0xb6cc bytes) 000562 1788202176.050189 trunc from 0x03ff55 to 0x01fa1e (0x20536 bytes) 000563 1788202176.221229 mapread 0x01b8c9 thru 0x01fa1e (0x4156 bytes) 000566 1788202176.245913 trunc from 0x01fa1f to 0x03a27c (0x1a85e bytes) 000567 1788202176.283028 mapread 0x032861 thru 0x03a27c (0x7a1c bytes) 000568 1788202176.311390 write 0x001f7a thru 0x0095f9 (0x7680 bytes) 000569 1788202176.341267 read 0x020da1 thru 0x0264dd (0x573d bytes) 000570 1788202176.352524 trunc from 0x03a27d to 0x034c41 (0x563b bytes) 000572 1788202176.372650 mapread 0x017fc8 thru 0x01a286 (0x22bf bytes) 000573 1788202176.373920 read 0x01ad83 thru 0x023b7b (0x8df9 bytes) 000574 1788202176.375552 write 0x021090 thru 0x02173b (0x6ac bytes) 000575 1788202176.377236 mapwrite 0x024c4e thru 0x02e89e (0x9c51 bytes) 000577 1788202176.528414 trunc from 0x034c42 to 0x01f541 (0x15700 bytes) 000578 1788202176.568542 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1f542): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f542): Mirror 8: 0x73e4bfd6 (stale) Mirror 9: 0x73e4bfd6 (stale) Mirror 10: 0x73e4bfd6 (stale) Mirror 11: 0x73e4bfd6 (stale) CRC-32 checksum value for chunk [0, 0x1f542): Mirror 1: 0x8c22e1d8 Mirror 12: 0x8c22e1d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f542: skipped 000579 1788202177.871358 write 0x0263bd thru 0x034d53 (0xe997 bytes) 000581 1788202177.972077 write 0x00ce01 thru 0x00d470 (0x670 bytes) 000586 1788202177.984029 mapwrite 0x024b0a thru 0x03053f (0xba36 bytes) 000587 1788202178.045829 trunc from 0x034d54 to 0x01ca91 (0x182c2 bytes) 000591 1788202178.069814 write 0x0318f6 thru 0x03a2a5 (0x89b0 bytes) 000592 1788202178.071682 mapread 0x0306da thru 0x0355c3 (0x4eea bytes) 000594 1788202178.079769 read 0x02e094 thru 0x02f6a0 (0x160d bytes) 000595 1788202178.080554 mapwrite 0x03cfd5 thru 0x03ffff (0x302b bytes) 000597 1788202178.160418 trunc from 0x040000 to 0x03454c (0xbab3 bytes) 000598 1788202178.186555 mapread 0x027851 thru 0x029c77 (0x2427 bytes) 000599 1788202178.199531 trunc from 0x03454d to 0x029ae9 (0xaa63 bytes) 000600 1788202178.226967 mapwrite 0x013260 thru 0x01f433 (0xc1d4 bytes) 000601 1788202178.370804 read 0x019129 thru 0x027cdf (0xebb7 bytes) 000606 1788202178.383975 mapwrite 0x0291b3 thru 0x030fc9 (0x7e17 bytes) 000608 1788202178.552971 mapread 0x0089dd thru 0x013163 (0xa787 bytes) 000610 1788202178.560369 mapread 0x01f4ef thru 0x021266 (0x1d78 bytes) 000612 1788202178.563281 mapread 0x014626 thru 0x017129 (0x2b04 bytes) 000613 1788202178.566652 mapwrite 0x037a1f thru 0x03ffff (0x85e1 bytes) 000615 1788202178.658662 write 0x03e151 thru 0x03ffff (0x1eaf bytes) 000616 1788202178.659734 write 0x01e6ab thru 0x02ddef (0xf745 bytes) 000617 1788202178.662742 write 0x018eb2 thru 0x0271e5 (0xe334 bytes) 000619 1788202178.666657 mapread 0x02071e thru 0x030045 (0xf928 bytes) 000620 1788202178.672320 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xb104a87b (stale) Mirror 9: 0xb104a87b (stale) Mirror 10: 0xb104a87b (stale) Mirror 11: 0xb104a87b (stale) Mirror 12: 0x7aa28f47 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xea2118bd Mirror 13: 0xea2118bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000621 1788202180.083615 write 0x0345b0 thru 0x034d82 (0x7d3 bytes) 000622 1788202180.239313 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xb104a87b (stale) Mirror 9: 0xb104a87b (stale) Mirror 10: 0xb104a87b (stale) Mirror 11: 0xb104a87b (stale) Mirror 12: 0x7aa28f47 (stale) Mirror 13: 0xea2118bd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b084ecc Mirror 14: 0x4b084ecc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000623 1788202182.050380 trunc from 0x040000 to 0x00148d (0x3eb72 bytes) 000625 1788202182.285169 mapwrite 0x012826 thru 0x01b0e9 (0x88c4 bytes) 000626 1788202182.552306 mapwrite 0x018d1a thru 0x0227a5 (0x9a8c bytes) 000630 1788202182.785165 mapwrite 0x03389a thru 0x03a32c (0x6a93 bytes) 000636 1788202182.994629 write 0x01d766 thru 0x02a007 (0xc8a2 bytes) 000641 1788202183.023785 trunc from 0x03a32d to 0x02fe25 (0xa507 bytes) 000642 1788202183.063728 mapread 0x00616d thru 0x00f862 (0x96f6 bytes) 000643 1788202183.083288 read 0x01e2ec thru 0x023383 (0x5098 bytes) 000645 1788202183.084539 trunc from 0x02fe26 to 0x02ecb9 (0x116c bytes) 000646 1788202183.108567 mapread 0x0244e1 thru 0x025520 (0x1040 bytes) 000647 1788202183.109746 write 0x01f73e thru 0x020fef (0x18b2 bytes) 000649 1788202183.119380 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ecba): Mirror 8: 0xb0ba4a22 (stale) Mirror 9: 0xb0ba4a22 (stale) Mirror 10: 0xb0ba4a22 (stale) Mirror 11: 0xb0ba4a22 (stale) Mirror 13: 0x22c43fc4 (stale) Mirror 14: 0x22c43fc4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ecba): Mirror 8: 0xb0ba4a22 (stale) Mirror 9: 0xb0ba4a22 (stale) Mirror 10: 0xb0ba4a22 (stale) Mirror 11: 0xb0ba4a22 (stale) Mirror 13: 0x22c43fc4 (stale) Mirror 14: 0x22c43fc4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ecba: skipped 000650 1788202183.678649 trunc from 0x02ecba to 0x00a276 (0x24a43 bytes) 000651 1788202183.710708 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xa277): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa277): Mirror 8: 0x661c5e47 (stale) Mirror 9: 0x661c5e47 (stale) Mirror 10: 0x661c5e47 (stale) Mirror 11: 0x661c5e47 (stale) Mirror 13: 0x237705b8 (stale) Mirror 14: 0x237705b8 (stale) CRC-32 checksum value for chunk [0, 0xa277): Mirror 1: 0x20533634 Mirror 15: 0x20533634 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa277: skipped 000652 1788202185.252422 mapread 0x007b83 thru 0x0087a2 (0xc20 bytes) 000653 1788202185.344103 MIRROR_RESYNC 8 mirrors 000654 1788202187.282965 read 0x0069dc thru 0x00a276 (0x389b bytes) 000655 1788202187.297930 trunc from 0x00a277 to 0x037928 (0x2d6b2 bytes) 000656 1788202187.401431 read 0x02695d thru 0x02adb6 (0x445a bytes) 000658 1788202187.418233 mapwrite 0x038529 thru 0x03ffff (0x7ad7 bytes) 000661 1788202187.514279 MIRROR_RESYNC 8 mirrors 000662 1788202190.553923 mapread 0x0090e0 thru 0x00f248 (0x6169 bytes) 000665 1788202190.607452 write 0x00fad5 thru 0x0118ab (0x1dd7 bytes) 000666 1788202190.693759 trunc from 0x040000 to 0x02133a (0x1ecc5 bytes) 000667 1788202190.723055 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2133b): Mirror 8: 0x934b5fb4 (stale) Mirror 10: 0x934b5fb4 (stale) Mirror 11: 0x934b5fb4 (stale) Mirror 13: 0x934b5fb4 (stale) Mirror 14: 0x934b5fb4 (stale) Mirror 15: 0x934b5fb4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2133b): Mirror 8: 0x934b5fb4 (stale) Mirror 10: 0x934b5fb4 (stale) Mirror 11: 0x934b5fb4 (stale) Mirror 13: 0x934b5fb4 (stale) Mirror 14: 0x934b5fb4 (stale) Mirror 15: 0x934b5fb4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2133b: skipped 000670 1788202191.177328 write 0x00c2c0 thru 0x01364a (0x738b bytes) 000671 1788202191.197033 mapwrite 0x0225d4 thru 0x02ff06 (0xd933 bytes) 000672 1788202191.352593 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ff07): Mirror 10: 0xe609dea (stale) Mirror 11: 0xe609dea (stale) Mirror 13: 0xe609dea (stale) Mirror 14: 0xe609dea (stale) Mirror 15: 0xe609dea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ff07): Mirror 10: 0xe609dea (stale) Mirror 11: 0xe609dea (stale) Mirror 13: 0xe609dea (stale) Mirror 14: 0xe609dea (stale) Mirror 15: 0xe609dea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ff07: skipped 000673 1788202191.953182 MIRROR_RESYNC 6 mirrors 000675 1788202193.388575 trunc from 0x02ff07 to 0x024c2b (0xb2db bytes) 000676 1788202193.586019 write 0x018484 thru 0x0225f5 (0xa172 bytes) 000678 1788202193.609154 trunc from 0x024c2c to 0x027ac9 (0x2e9e bytes) 000680 1788202193.666535 mapwrite 0x019527 thru 0x026c65 (0xd73f bytes) 000682 1788202193.828164 mapread 0x0026ab thru 0x010bf3 (0xe549 bytes) 000683 1788202193.844230 mapwrite 0x01353a thru 0x015f5a (0x2a21 bytes) 000692 1788202194.012615 mapread 0x026f83 thru 0x027ac9 (0xb47 bytes) 000693 1788202194.016059 mapwrite 0x03c582 thru 0x03ffff (0x3a7e bytes) 000695 1788202194.181600 read 0x033bf0 thru 0x03ffff (0xc410 bytes) 000696 1788202194.187947 mapread 0x02549e thru 0x02cc16 (0x7779 bytes) 000698 1788202194.200152 mapread 0x011a79 thru 0x019d8f (0x8317 bytes) 000699 1788202194.201614 mapwrite 0x009d11 thru 0x009edc (0x1cc bytes) 000703 1788202194.269340 trunc from 0x040000 to 0x024076 (0x1bf89 bytes) 000707 1788202194.310697 trunc from 0x024077 to 0x01866f (0xba07 bytes) 000709 1788202194.353506 mapwrite 0x02eb10 thru 0x0327a8 (0x3c99 bytes) 000713 1788202194.516582 trunc from 0x0327a9 to 0x00fd34 (0x22a74 bytes) 000715 1788202194.556711 MIRROR_RESYNC 6 mirrors 000717 1788202196.179618 mapwrite 0x016542 thru 0x01e0ea (0x7ba9 bytes) 000718 1788202196.415377 write 0x039c77 thru 0x03baa2 (0x1e2c bytes) 000719 1788202196.424566 mapread 0x02d7a5 thru 0x03b026 (0xd882 bytes) 000720 1788202196.457021 mapwrite 0x03e16d thru 0x03ffff (0x1e93 bytes) 000721 1788202196.625119 read 0x015fcb thru 0x016929 (0x95f bytes) 000722 1788202196.630598 trunc from 0x040000 to 0x031ec1 (0xe13e bytes) 000723 1788202196.674271 trunc from 0x031ec2 to 0x0256c2 (0xc7ff bytes) 000724 1788202196.706652 mapread 0x00a5eb thru 0x00dd75 (0x378b bytes) 000725 1788202196.721454 mapread 0x023a6d thru 0x0256c2 (0x1c56 bytes) 000726 1788202196.747742 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x256c3): Mirror 10: 0xf42d1e40 (stale) Mirror 11: 0xf42d1e40 (stale) Mirror 13: 0xf42d1e40 (stale) Mirror 14: 0xf42d1e40 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x256c3): Mirror 10: 0xf42d1e40 (stale) Mirror 11: 0xf42d1e40 (stale) Mirror 13: 0xf42d1e40 (stale) Mirror 14: 0xf42d1e40 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x256c3: skipped 000728 1788202197.326803 trunc from 0x0256c3 to 0x0107b0 (0x14f12 bytes) 000729 1788202197.366884 mapwrite 0x02ab6f thru 0x032720 (0x7bb2 bytes) 000730 1788202197.534502 read 0x020438 thru 0x023420 (0x2fe9 bytes) 000732 1788202197.543385 read 0x014015 thru 0x01b9e4 (0x79d0 bytes) 000735 1788202197.586175 read 0x001de0 thru 0x010a23 (0xec44 bytes) 000740 1788202197.623545 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32721): Mirror 10: 0xfab2d8ea (stale) Mirror 11: 0xfab2d8ea (stale) Mirror 13: 0xfab2d8ea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32721): Mirror 10: 0xfab2d8ea (stale) Mirror 11: 0xfab2d8ea (stale) Mirror 13: 0xfab2d8ea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32721: skipped 000746 1788202198.238069 write 0x019350 thru 0x01e0fe (0x4daf bytes) 000748 1788202198.260370 mapread 0x011862 thru 0x01e5d1 (0xcd70 bytes) 000751 1788202198.324335 trunc from 0x032721 to 0x02a2d0 (0x8450 bytes) 000752 1788202198.359005 write 0x033bf8 thru 0x035813 (0x1c1c bytes) 000755 1788202198.378069 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35814): Mirror 10: 0xc501eb0c (stale) Mirror 13: 0xc501eb0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35814): Mirror 10: 0xc501eb0c (stale) Mirror 13: 0xc501eb0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35814: skipped 000757 1788202199.035628 read 0x0216fe thru 0x0230b3 (0x19b6 bytes) 000758 1788202199.062179 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35814): Mirror 13: 0xc501eb0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35814): Mirror 13: 0xc501eb0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35814: skipped 000759 1788202199.592703 mapwrite 0x03f8af thru 0x03ffff (0x751 bytes) 000760 1788202199.655199 trunc from 0x040000 to 0x01ff97 (0x20068 bytes) 000761 1788202199.678608 mapread 0x003268 thru 0x00c3b0 (0x9149 bytes) 000762 1788202199.701555 trunc from 0x01ff98 to 0x00d115 (0x12e82 bytes) 000763 1788202199.720678 trunc from 0x00d116 to 0x02dfbe (0x20ea9 bytes) 000768 1788202199.759371 mapread 0x01fcf5 thru 0x022d79 (0x3085 bytes) 000770 1788202199.774561 read 0x029d45 thru 0x02a841 (0xafd bytes) 000771 1788202199.781203 write 0x01c19f thru 0x01f3ec (0x324e bytes) 000772 1788202199.783559 mapread 0x01008c thru 0x014666 (0x45db bytes) 000773 1788202199.795814 trunc from 0x02dfbf to 0x03dd62 (0xfda4 bytes) 000774 1788202199.816051 read 0x0055a1 thru 0x00f30c (0x9d6c bytes) 000775 1788202199.817569 mapread 0x01426a thru 0x01669e (0x2435 bytes) 000776 1788202199.820601 write 0x03b135 thru 0x03ffff (0x4ecb bytes) 000779 1788202199.835714 write 0x03ea45 thru 0x03ffff (0x15bb bytes) 000780 1788202199.838612 mapwrite 0x019244 thru 0x0214bd (0x827a bytes) 000782 1788202199.895418 write 0x03fb2c thru 0x03ffff (0x4d4 bytes) 000783 1788202199.898017 trunc from 0x040000 to 0x02fbbe (0x10441 bytes) 000784 1788202199.917684 mapread 0x02d65e thru 0x02fbbe (0x2561 bytes) 000785 1788202199.920453 trunc from 0x02fbbf to 0x01de6d (0x11d51 bytes) 000786 1788202199.947446 mapwrite 0x03cb8f thru 0x03ffff (0x3471 bytes) 000787 1788202200.039676 mapwrite 0x00e5d9 thru 0x01bc0a (0xd632 bytes) 000788 1788202200.101369 mapread 0x011cef thru 0x017582 (0x5894 bytes) 000789 1788202200.102608 mapread 0x000236 thru 0x0011fc (0xfc7 bytes) 000790 1788202200.118557 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000793 1788202200.293784 write 0x00cd18 thru 0x00d7b1 (0xa9a bytes) 000794 1788202200.312720 read 0x039474 thru 0x03a185 (0xd12 bytes) 000799 1788202200.337112 mapwrite 0x0231a1 thru 0x02c7e5 (0x9645 bytes) 000802 1788202200.481944 MIRROR_EXTEND to 2 mirrors 000803 1788202201.404598 write 0x02464d thru 0x029ccd (0x5681 bytes) 000804 1788202201.591695 read 0x0368ee thru 0x03ffff (0x9712 bytes) 000806 1788202201.609057 read 0x02a18a thru 0x02b90d (0x1784 bytes) 000807 1788202201.620627 mapwrite 0x03a180 thru 0x03e32a (0x41ab bytes) 000812 1788202201.690105 mapread 0x03165a thru 0x03867d (0x7024 bytes) 000813 1788202201.692572 mapread 0x017548 thru 0x01b664 (0x411d bytes) 000814 1788202201.694426 trunc from 0x040000 to 0x020f24 (0x1f0db bytes) 000816 1788202201.785716 write 0x02ea32 thru 0x03a275 (0xb844 bytes) 000817 1788202201.804274 MIRROR_RESYNC 2 mirrors 000818 1788202202.870118 trunc from 0x03a276 to 0x00d48c (0x2cde9 bytes) 000819 1788202203.004699 read 0x00d14a thru 0x00d48c (0x343 bytes) 000820 1788202203.013777 mapwrite 0x01e6fb thru 0x026954 (0x825a bytes) 000822 1788202203.132271 read 0x021253 thru 0x026954 (0x5702 bytes) 000823 1788202203.145394 MIRROR_RESYNC 2 mirrors 000824 1788202204.034573 write 0x00a40c thru 0x00f0f7 (0x4cec bytes) 000825 1788202204.178349 trunc from 0x026955 to 0x03b5cd (0x14c79 bytes) 000826 1788202204.258827 write 0x03eebb thru 0x03ffff (0x1145 bytes) 000829 1788202204.265596 read 0x006556 thru 0x0162a4 (0xfd4f bytes) 000830 1788202204.299168 mapread 0x01b689 thru 0x020fad (0x5925 bytes) 000834 1788202204.330547 mapread 0x03c233 thru 0x03ffff (0x3dcd bytes) 000835 1788202204.354637 mapread 0x02b2c8 thru 0x038325 (0xd05e bytes) 000836 1788202204.406631 mapread 0x005a40 thru 0x01502d (0xf5ee bytes) 000837 1788202204.445933 read 0x01b56c thru 0x01f2fc (0x3d91 bytes) 000838 1788202204.448780 mapwrite 0x022f61 thru 0x023e07 (0xea7 bytes) 000840 1788202204.556468 trunc from 0x040000 to 0x031d17 (0xe2e8 bytes) 000841 1788202204.626670 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31d18): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31d18): Mirror 2: 0x7e06b301 (stale) CRC-32 checksum value for chunk [0, 0x31d18): Mirror 1: 0xd407df00 Mirror 3: 0xd407df00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31d18: skipped 000842 1788202206.050664 trunc from 0x031d18 to 0x023b4d (0xe1ca bytes) 000843 1788202206.201471 write 0x033cf0 thru 0x03e084 (0xa395 bytes) 000846 1788202206.215666 read 0x03ced2 thru 0x03e084 (0x11b3 bytes) 000848 1788202206.219550 mapwrite 0x01a973 thru 0x021718 (0x6da6 bytes) 000850 1788202206.397087 mapread 0x03a4ba thru 0x03e084 (0x3bcb bytes) 000851 1788202206.398473 mapread 0x028b44 thru 0x032c8e (0xa14b bytes) 000852 1788202206.447431 write 0x02bd97 thru 0x02c725 (0x98f bytes) 000856 1788202206.473090 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e085): Mirror 3: 0x13e21ec8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e085): Mirror 3: 0x13e21ec8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e085: skipped 000857 1788202206.956762 mapread 0x00af9d thru 0x0169bd (0xba21 bytes) 000859 1788202207.051381 mapread 0x02f119 thru 0x032e0a (0x3cf2 bytes) 000860 1788202207.081597 mapread 0x038c47 thru 0x03e084 (0x543e bytes) 000862 1788202207.127628 mapwrite 0x034617 thru 0x03fd5c (0xb746 bytes) 000863 1788202207.308795 read 0x010a9e thru 0x014bb8 (0x411b bytes) 000864 1788202207.313736 read 0x03c74f thru 0x03fd5c (0x360e bytes) 000865 1788202207.321391 write 0x018182 thru 0x01cda2 (0x4c21 bytes) 000868 1788202207.343258 read 0x0063fd thru 0x00f5eb (0x91ef bytes) 000869 1788202207.371433 MIRROR_RESYNC 2 mirrors 000870 1788202208.437404 MIRROR_RESYNC 2 mirrors 000871 1788202208.818973 write 0x003790 thru 0x007119 (0x398a bytes) 000872 1788202208.972572 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 3: 0x278d9ae0 (stale) CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 1: 0x5b12b1c4 Mirror 4: 0x5b12b1c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd5d: skipped 000873 1788202210.408820 read 0x03a9d0 thru 0x03fd5c (0x538d bytes) 000874 1788202210.424379 mapwrite 0x026c6d thru 0x02e135 (0x74c9 bytes) 000876 1788202210.613276 write 0x01529d thru 0x01e696 (0x93fa bytes) 000877 1788202210.653571 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 3: 0x278d9ae0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd5d): Mirror 3: 0x278d9ae0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd5d: skipped 000878 1788202210.985736 MIRROR_RESYNC 2 mirrors 000879 1788202211.903955 trunc from 0x03fd5d to 0x02bb84 (0x141d8 bytes) 000882 1788202212.035529 trunc from 0x02bb85 to 0x01bfc7 (0xfbbd bytes) 000883 1788202212.056711 trunc from 0x01bfc8 to 0x03279b (0x167d4 bytes) 000884 1788202212.076472 trunc from 0x03279c to 0x0178d8 (0x1aec3 bytes) 000885 1788202212.092206 mapwrite 0x02b8dd thru 0x03a05f (0xe783 bytes) 000886 1788202212.197027 mapread 0x00f384 thru 0x01da8a (0xe707 bytes) 000887 1788202212.231063 mapwrite 0x01ca8f thru 0x028d7f (0xc2f1 bytes) 000891 1788202212.299777 read 0x033374 thru 0x033902 (0x58f bytes) 000892 1788202212.303344 read 0x02eb77 thru 0x033d6a (0x51f4 bytes) 000894 1788202212.309684 mapwrite 0x032e4d thru 0x036f1f (0x40d3 bytes) 000895 1788202212.388423 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a060): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a060): Mirror 3: 0x1715c74d (stale) CRC-32 checksum value for chunk [0, 0x3a060): Mirror 1: 0xbcda42a1 Mirror 4: 0xbcda42a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a060: skipped 000896 1788202213.404678 write 0x03326d thru 0x036650 (0x33e4 bytes) 000897 1788202213.522539 read 0x033eee thru 0x036d55 (0x2e68 bytes) 000898 1788202213.523530 read 0x01b1c1 thru 0x01be39 (0xc79 bytes) 000899 1788202213.534704 mapwrite 0x00ed15 thru 0x012927 (0x3c13 bytes) 000900 1788202213.585548 mapwrite 0x00ee75 thru 0x00eec5 (0x51 bytes) 000902 1788202213.651406 read 0x013abc thru 0x02353a (0xfa7f bytes) 000903 1788202213.662532 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a060): Mirror 3: 0x1715c74d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a060): Mirror 3: 0x1715c74d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a060: skipped 000906 1788202214.174374 read 0x00ce26 thru 0x00d77d (0x958 bytes) 000908 1788202214.194585 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000911 1788202214.613970 mapread 0x028f69 thru 0x02c3f3 (0x348b bytes) 000913 1788202214.717079 mapread 0x02f54d thru 0x02f91b (0x3cf bytes) 000914 1788202214.721514 trunc from 0x03a060 to 0x016983 (0x236dc bytes) 000915 1788202214.852506 MIRROR_EXTEND to 2 mirrors 000916 1788202216.312538 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000917 1788202216.559915 read 0x00f78e thru 0x016983 (0x71f6 bytes) 000918 1788202216.574988 mapread 0x002791 thru 0x00c43c (0x9cac bytes) 000919 1788202216.638387 trunc from 0x016984 to 0x027d24 (0x113a1 bytes) 000921 1788202216.678633 mapread 0x02524b thru 0x026ad2 (0x1888 bytes) 000922 1788202216.692026 mapread 0x015d0c thru 0x018d91 (0x3086 bytes) 000923 1788202216.701237 read 0x014a3b thru 0x01dc48 (0x920e bytes) 000927 1788202216.706670 mapread 0x00b810 thru 0x01856b (0xcd5c bytes) 000928 1788202216.737719 write 0x03b5c1 thru 0x03ffff (0x4a3f bytes) 000929 1788202216.740779 write 0x034224 thru 0x03eb75 (0xa952 bytes) 000930 1788202216.767022 trunc from 0x040000 to 0x03b61f (0x49e0 bytes) 000932 1788202216.800991 write 0x01003a thru 0x013cad (0x3c74 bytes) 000934 1788202216.802706 write 0x01ebb5 thru 0x02b249 (0xc695 bytes) 000935 1788202216.808383 write 0x00b248 thru 0x01a236 (0xefef bytes) 000937 1788202216.818321 mapread 0x02e450 thru 0x03a406 (0xbfb7 bytes) 000939 1788202216.823622 read 0x0364a3 thru 0x03aecf (0x4a2d bytes) 000940 1788202216.827247 read 0x003a65 thru 0x009875 (0x5e11 bytes) 000942 1788202216.831478 trunc from 0x03b620 to 0x02ada6 (0x10879 bytes) 000944 1788202216.883692 mapwrite 0x02d007 thru 0x03a09e (0xd098 bytes) 000945 1788202217.078778 mapread 0x003c10 thru 0x006b12 (0x2f03 bytes) 000946 1788202217.079952 read 0x0171e9 thru 0x01cbe9 (0x5a01 bytes) 000947 1788202217.099796 MIRROR_EXTEND to 2 mirrors 000950 1788202218.346875 mapwrite 0x021a48 thru 0x024277 (0x2830 bytes) 000953 1788202218.680388 mapread 0x03145d thru 0x038d3f (0x78e3 bytes) 000965 1788202218.723525 read 0x024f84 thru 0x02e997 (0x9a14 bytes) 000967 1788202218.730653 trunc from 0x03a09f to 0x03f266 (0x51c8 bytes) 000970 1788202218.779655 mapwrite 0x00c9bf thru 0x017fa3 (0xb5e5 bytes) 000971 1788202218.927946 mapread 0x004ec7 thru 0x0131c5 (0xe2ff bytes) 000972 1788202218.960289 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x18f:0x0]: Invalid argument (22) 000980 1788202219.227371 read 0x0170c0 thru 0x021fcc (0xaf0d bytes) 000981 1788202219.238244 mapread 0x01a857 thru 0x0216de (0x6e88 bytes) 000982 1788202219.241048 mapread 0x009485 thru 0x00fb32 (0x66ae bytes) 000983 1788202219.290067 write 0x039063 thru 0x03ffff (0x6f9d bytes) 000984 1788202219.299623 read 0x00fa3d thru 0x01ca99 (0xd05d bytes) 000985 1788202219.311456 write 0x02dae9 thru 0x03cbf7 (0xf10f bytes) 000986 1788202219.313770 read 0x02539b thru 0x028d5d (0x39c3 bytes) 000987 1788202219.315584 mapwrite 0x039647 thru 0x03d125 (0x3adf bytes) 000988 1788202219.406505 write 0x021e29 thru 0x028dc3 (0x6f9b bytes) 000989 1788202219.413206 mapwrite 0x00e973 thru 0x0109a8 (0x2036 bytes) 000990 1788202219.523376 write 0x03ffbc thru 0x03ffff (0x44 bytes) 000994 1788202219.527592 mapwrite 0x0274dd thru 0x032972 (0xb496 bytes) 000995 1788202219.654409 mapwrite 0x007f83 thru 0x01243e (0xa4bc bytes) 000998 1788202219.760424 mapwrite 0x035609 thru 0x03ffff (0xa9f7 bytes) 001000 1788202219.935545 write 0x00f524 thru 0x01b6aa (0xc187 bytes) All operations completed A-OK! PASS 70a (107s) == sanity-flr test 71: check mirror extend parent fid ==== 14:50:25 (1788202225) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 14:50:29 (1788202229) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.286354 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21089 s, 5.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b5:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2b4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ba:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 14:50:39 (1788202239) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8690112 .. Extending file size to 4628608 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6217440 .. failed Extending file size to 8145632 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8418176 .. Extending file size to 9211776 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9296832 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8670272 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57381 57382 57385 57386 57388 done 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) 192.168.203.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (89s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:52:07 (1788202327) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2764416 .. Extending file size to 1683680 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2296928 .. Extending file size to 8565760 .. Extending file size to 7458560 .. Extending file size to 7494304 .. Extending file size to 8368992 .. Extending file size to 8673280 .. Extending file size to 9200032 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 58904 58906 58907 58909 58911 done 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) 192.168.203.156@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (88s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:53:35 (1788202415) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161089 s, 13.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.499549 s, 4.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xc788f2d5 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:53:58 (1788202438) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12487 s, 16.8 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 14:54:06 (1788202446) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d8:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:54:14 (1788202454) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.272849 s, 7.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:54:23 (1788202463) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.292142 s, 7.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d3:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:54:32 (1788202472) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.271324 s, 7.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d5:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:55:06 (1788202506) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg356-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223204 s, 4.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d2:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:56:10 (1788202570) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg356-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.40267 s, 5.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d3:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:57:06 (1788202626) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg356-server mds-ost sync done. Creating new pool test_204f oleg356-server: Pool lustre.test_204f created Adding targets to pool oleg356-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg356-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.274364 s, 7.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg356-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg356-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg356-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:58:20 (1788202700) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:58:29 (1788202709) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:58:37 (1788202717) oleg356-server: Pool lustre.test_206 created Adding targets to pool oleg356-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg356-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg356-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg356-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg356-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:59:03 (1788202743) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14473 s, 7.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568919 s, 18.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x940e7ee6 Mirror 3: 0x940e7ee6 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:59:15 (1788202755) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:59:19 (1788202759) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:59:22 (1788202762) Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.131762 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1c50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1b40000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1b40000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1b40000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x22b0000 Read this (4 bytes): BBBB 192.168.203.156@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg356-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:59:58 (1788202798) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0486895 s, 168 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:00:05 (1788202805) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554338 s, 18.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fd lmm_fid: [0x200000404:0x1fd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1fd lmm_fid: [0x200000404:0x1fd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e3:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834429 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 15:00:06.000000000 -0400 Modify: 2026-08-31 15:00:06.000000000 -0400 Change: 2026-08-31 15:00:08.000000000 -0400 Birth: 2026-08-31 15:00:07.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:00:13 (1788202813) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:00:15 (1788202815) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.564267 s, 72.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0506026 s, 80.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 31 15:00 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ff:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 31 15:00 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:00:24 (1788202824) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:00:31 (1788202831) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1755 sec =========== 15:00:41 (1788202841) === sanity-flr: start cleanup 15:00:43 (1788202843) === === sanity-flr: finish cleanup 15:00:51 (1788202851) ===