-----============= acceptance-small: sanity-flr ============----- Mon Sep 14 12:09:39 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg130-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:09:58 (1789402198) === oleg130-client.virtnet: executing check_config_client /mnt/lustre oleg130-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg130-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9447452e0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9447452e0000.idle_timeout=debug disable quota as required oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:10:29 (1789402229) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7638] == sanity-flr test 0a: lfs mirror create with -N option == 12:10:31 (1789402231) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:10:47 (1789402247) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:10:50 (1789402250) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:10:54 (1789402254) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:11:17 (1789402277) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:11:21 (1789402281) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 12:11:24 (1789402284) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9447452e0000.stats=clear osc.lustre-OST0001-osc-ffff9447452e0000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:11:31 (1789402291) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:11:42 (1789402302) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:11:51 (1789402311) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:11:54 (1789402314) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:12:01 (1789402321) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 12:12:11 (1789402331) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:12:17 (1789402337) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:12:20 (1789402340) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:12:26 (1789402346) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:12:36 (1789402356) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:12:42 (1789402362) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 2.26505 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153638 s, 6.8 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:12:53 (1789402373) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0814521 s, 51.5 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:13:00 (1789402380) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:13:11 (1789402391) Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9447452e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9447452e0000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9447452e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9447452e0000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg130-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:14:27 (1789402467) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.316931 s, 16.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9447452e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9447452e0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (44s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:15:11 (1789402511) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9447452e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9447452e0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9447452e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9447452e0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg130-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:16:32 (1789402592) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.35199 s, 9.8 MB/s -rw-r--r-- 1 root root 52428800 Sep 14 12:16 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.989s user 0m0.001s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9447452e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9447452e0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.339s user 0m0.001s sys 0m0.197s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg130-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg130-server oleg130-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9447452e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9447452e0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.294s user 0m0.001s sys 0m0.163s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg130-server: oleg130-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg130-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg130-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9447452e0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9447452e0000.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:18:09 (1789402689) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:18:12 (1789402692) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:18:15 (1789402695) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 12:18:19 (1789402699) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 12:18:26 (1789402706) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.459592 s, 27.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:18:37 (1789402717) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163441 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248614 s, 42.2 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:18:43 (1789402723) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:18:51 (1789402731) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00116981 s, 875 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00219945 s, 466 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000743601 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00118746 s, 862 kB/s flv=10 olv=10 PASS 36d (13s) == sanity-flr test 37: mirror I/O API verification ======= 12:19:04 (1789402744) create test files with size 15751467 .. c0d654ac8aa3666fa364222b536580c1 - 7c50258b1425a1961e1e986bbb3f5c2f - 83267cd5a215a28497b311a438a4b898 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg130-client.virtnet /mnt/lustre (opts:) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre PASS 37 (29s) == sanity-flr test 38: resync ============================ 12:19:33 (1789402773) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:19:50 (1789402790) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:19:57 (1789402797) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111373 s, 9.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546113 s, 19.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 12:20:06 (1789402806) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134554 s, 15.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.309885 s, 13.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137143 s, 15.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 12:20:21 (1789402821) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 12:20:24 (1789402824) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:20:27 (1789402827) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418203 s, 25.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 12:20:35 (1789402835) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:20:39 (1789402839) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:20:48 (1789402848) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.246861 s, 42.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg130-client.virtnet /mnt/lustre (opts:) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:20:58 (1789402858) llite.lustre-ffff94475862b000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.304342 s, 34.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg130-client.virtnet /mnt/lustre (opts:) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9447452e2800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:21:11 (1789402871) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:21:49 (1789402909) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:22:25 (1789402945) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 12:22:29 (1789402949) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.22667 s, 13.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.118697 s, 26.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xded26dab (nosync,stale) Mirror 2: 0xded26dab (nosync,stale) ** verify mirror 2 doesn't change original checksum: fb1d5d71ef2f61248642444f2650c510 - mirror 1 checksum: 6c7a079b03a106c2dc17c2c9b2548094 - mirror 2 checksum: fb1d5d71ef2f61248642444f2650c510 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-14T12:22:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-14T12:22:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: fb1d5d71ef2f61248642444f2650c510 - mirror 1 checksum: 6c7a079b03a106c2dc17c2c9b2548094 - mirror 2 checksum: 6c7a079b03a106c2dc17c2c9b2548094 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-14T12:22:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-14T12:22:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xed:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:22:40 (1789402960) Failing mds1 on oleg130-server Stopping /mnt/lustre-mds1 (opts:) on oleg130-server 12:22:52 (1789402972) shut down facet: mds1 facet_host: oleg130-server facet_failover_host: oleg130-server Failover mds1 to oleg130-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg130-server: oleg130-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-MDT0000 12:23:15 (1789402995) targets are mounted 12:23:15 (1789402995) facet_failover done oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644009 s, 16.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xee:0x0] } PASS 50A (62s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:23:42 (1789403022) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:23:46 (1789403026) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:23:49 (1789403029) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00720403 s, 2.3 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:23:58 (1789403038) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:24:02 (1789403042) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.043776 s, 93.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0870434 s, 47.1 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:24:11 (1789403051) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:24:33 (1789403073) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:25:00 (1789403100) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:25:21 (1789403121) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:25:46 (1789403146) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 12:25:54 (1789403154) mirror create pid 54329 mirror split pid 54330 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54329 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54330 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 12:27:01 (1789403221) Seed set to 3223 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x2d5ca 000001 1789403223.272742 trunc from 00000000 to 0x02d5c9 (0x2d5ca bytes) 000002 1789403223.307467 mapwrite 0x02ec6d thru 0x03d9b9 (0xed4d bytes) 000003 1789403223.429691 write 0x035583 thru 0x0366fa (0x1178 bytes) 000004 1789403223.430564 mapwrite 0x03328d thru 0x03fc86 (0xc9fa bytes) 000005 1789403223.502723 trunc from 0x03fc87 to 0x012b0e (0x2d178 bytes) 000006 1789403223.517622 read 0x00a693 thru 0x0114d6 (0x6e44 bytes) 000009 1789403223.528030 mapread 0x000d9e thru 0x00967a (0x88dd bytes) 000010 1789403223.534754 write 0x014447 thru 0x01cff2 (0x8bac bytes) 000011 1789403223.536513 read 0x01be28 thru 0x01cff2 (0x11cb bytes) 000012 1789403223.537543 read 0x006668 thru 0x0121c5 (0xbb5e bytes) truncating to largest ever: 0x2dcef 000014 1789403223.543315 trunc from 0x01cff3 to 0x02dcee (0x10cfc bytes) 000016 1789403223.560114 read 0x008233 thru 0x018210 (0xffde bytes) 000017 1789403223.562328 mapwrite 0x00faf0 thru 0x011a31 (0x1f42 bytes) 000020 1789403223.608214 mapread 0x007b87 thru 0x008742 (0xbbc bytes) 000024 1789403223.612288 trunc from 0x02dcef to 0x011b9e (0x1c150 bytes) 000025 1789403223.630991 mapread 0x0099df thru 0x011b9e (0x81c0 bytes) 000026 1789403223.632518 write 0x032674 thru 0x03ffff (0xd98c bytes) 000027 1789403223.635206 mapread 0x012590 thru 0x018e97 (0x6908 bytes) 000029 1789403223.642631 punch from 0x015b3e to 0x020a4b (0xaf0e bytes) 000030 1789403223.654352 trunc from 0x040000 to 0x017c5d (0x283a2 bytes) 000031 1789403223.673447 trunc from 0x017c5e to 0x016829 (0x1434 bytes) 000032 1789403223.690170 MIRROR_EXTEND to 2 mirrors 000033 1789403224.353632 mapread 0x003105 thru 0x01234d (0xf249 bytes) 000035 1789403224.394447 mapwrite 0x0119eb thru 0x020282 (0xe898 bytes) 000036 1789403224.678079 mapread 0x001752 thru 0x00dcc0 (0xc56f bytes) 000037 1789403224.687143 read 0x0176a3 thru 0x020282 (0x8be0 bytes) 000038 1789403224.687947 trunc from 0x020283 to 0x0116fc (0xeb86 bytes) 000039 1789403224.711328 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x116fd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x116fd): Mirror 2: 0xb52e92e2 (stale) CRC-32 checksum value for chunk [0, 0x116fd): Mirror 1: 0xb52e92e2 Mirror 3: 0xb52e92e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x116fd: skipped 000040 1789403225.614254 read 0x007583 thru 0x00e086 (0x6b04 bytes) 000042 1789403225.627758 punch from 0x0058c0 to 0x00cced (0x742e bytes) 000043 1789403225.716041 read 0x00a8a9 thru 0x0116fc (0x6e54 bytes) 000044 1789403225.725392 mapread 0x0013f0 thru 0x002e7e (0x1a8f bytes) 000045 1789403225.726238 read 0x00a703 thru 0x0116fc (0x6ffa bytes) 000046 1789403225.726767 mapread 0x00d741 thru 0x00fba3 (0x2463 bytes) 000047 1789403225.727493 write 0x03d6bb thru 0x03e109 (0xa4f bytes) 000048 1789403225.729404 write 0x031543 thru 0x0334ef (0x1fad bytes) truncating to largest ever: 0x3fd7c 000051 1789403225.741269 trunc from 0x03e10a to 0x03fd7b (0x1c72 bytes) 000052 1789403225.758170 mapread 0x006e48 thru 0x0093ce (0x2587 bytes) 000053 1789403225.759186 trunc from 0x03fd7c to 0x00a25e (0x35b1d bytes) 000056 1789403225.782482 read 0x006e43 thru 0x00a25e (0x341c bytes) 000057 1789403225.783374 mapread 0x005acd thru 0x00a25e (0x4792 bytes) 000058 1789403225.786434 trunc from 0x00a25f to 0x00c888 (0x262a bytes) 000059 1789403225.802441 trunc from 0x00c889 to 0x03d16b (0x308e3 bytes) 000060 1789403225.816281 write 0x016ddf thru 0x02588d (0xeaaf bytes) 000061 1789403225.828233 write 0x023ec4 thru 0x024b71 (0xcae bytes) 000062 1789403225.829443 mapwrite 0x00feb3 thru 0x019bdc (0x9d2a bytes) 000064 1789403225.882159 trunc from 0x03d16c to 0x0002ec (0x3ce7f bytes) 000065 1789403225.900212 read 0x000087 thru 0x0002ec (0x266 bytes) 000066 1789403225.902119 mapread 0x00006b thru 0x0002ec (0x282 bytes) 000067 1789403225.903500 punch from 0x000259 to 0x0002ec (0x94 bytes) 000068 1789403225.908429 mapwrite 0x037142 thru 0x03ca63 (0x5922 bytes) 000070 1789403226.009597 read 0x022264 thru 0x02a5e6 (0x8383 bytes) 000071 1789403226.010080 mapwrite 0x036433 thru 0x03a8b4 (0x4482 bytes) 000075 1789403226.073537 punch from 0x02755d to 0x032ea7 (0xb94b bytes) 000076 1789403226.078394 read 0x0360f9 thru 0x0368cf (0x7d7 bytes) 000077 1789403226.079189 read 0x03a3a5 thru 0x03ca63 (0x26bf bytes) 000078 1789403226.080216 mapwrite 0x023c5e thru 0x029816 (0x5bb9 bytes) 000080 1789403226.140094 punch from 0x010ece to 0x01d0c6 (0xc1f9 bytes) 000081 1789403226.148574 MIRROR_RESYNC 3 mirrors 000083 1789403227.143071 MIRROR_RESYNC 3 mirrors 000084 1789403227.478468 mapwrite 0x034117 thru 0x03ac52 (0x6b3c bytes) 000085 1789403227.588479 read 0x017016 thru 0x022aaf (0xba9a bytes) 000087 1789403227.595867 write 0x020e0c thru 0x021f4c (0x1141 bytes) 000089 1789403227.597673 mapread 0x02d71e thru 0x03763d (0x9f20 bytes) 000090 1789403227.604375 read 0x00a438 thru 0x019ceb (0xf8b4 bytes) 000092 1789403227.619896 MIRROR_RESYNC 3 mirrors 000093 1789403228.469768 mapread 0x03afb0 thru 0x03ca63 (0x1ab4 bytes) 000097 1789403228.488097 write 0x036bfc thru 0x039d0d (0x3112 bytes) 000099 1789403228.568549 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ca64): Mirror 3: 0x9b825917 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca64): Mirror 3: 0x9b825917 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca64: skipped 000100 1789403228.854811 trunc from 0x03ca64 to 0x01fff6 (0x1ca6d bytes) 000101 1789403228.877340 mapread 0x00fd79 thru 0x01ae6e (0xb0f6 bytes) 000102 1789403228.912603 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1fff7): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fff7): Mirror 3: 0xb60b72f2 (stale) CRC-32 checksum value for chunk [0, 0x1fff7): Mirror 1: 0xb60b72f2 Mirror 4: 0xb60b72f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fff7: skipped 000105 1789403229.757382 trunc from 0x01fff7 to 0x0367eb (0x167f5 bytes) 000106 1789403229.840429 write 0x0125f9 thru 0x014bd5 (0x25dd bytes) 000107 1789403229.865063 MIRROR_RESYNC 3 mirrors 000108 1789403230.867997 punch from 0x01db12 to 0x022ab6 (0x4fa5 bytes) 000109 1789403230.995400 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x367ec): Mirror 3: 0x2da29bbc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367ec): Mirror 3: 0x2da29bbc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367ec: skipped 000110 1789403231.438400 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x367ec): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367ec): Mirror 3: 0x2da29bbc (stale) CRC-32 checksum value for chunk [0, 0x367ec): Mirror 1: 0x2da29bbc Mirror 4: 0x2da29bbc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367ec: skipped 000112 1789403232.629539 mapwrite 0x030157 thru 0x031643 (0x14ed bytes) 000113 1789403232.822503 punch from 0x027228 to 0x0273a6 (0x17f bytes) 000114 1789403232.845554 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x367ec): Mirror 4: 0x2da29bbc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367ec): Mirror 4: 0x2da29bbc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367ec: skipped 000115 1789403233.360430 mapread 0x01fbf9 thru 0x020b64 (0xf6c bytes) 000116 1789403233.384447 write 0x001212 thru 0x006ae7 (0x58d6 bytes) 000117 1789403233.425396 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000119 1789403233.594866 punch from 0x02897c to 0x0302f1 (0x7976 bytes) 000120 1789403233.600509 punch from 0x0010d8 to 0x006c69 (0x5b92 bytes) 000121 1789403233.604571 punch from 0x017942 to 0x022938 (0xaff7 bytes) 000122 1789403233.609838 trunc from 0x0367ec to 0x00c884 (0x29f67 bytes) 000123 1789403233.625395 punch from 0x000130 to 0x00c884 (0xc755 bytes) 000124 1789403233.629536 mapwrite 0x010ab2 thru 0x0166a4 (0x5bf3 bytes) 000126 1789403233.725764 punch from 0x0032d9 to 0x00b019 (0x7d41 bytes) 000127 1789403233.731550 read 0x0059a0 thru 0x0092b2 (0x3913 bytes) 000129 1789403233.747469 MIRROR_EXTEND to 2 mirrors 000130 1789403234.696095 mapwrite 0x031bfe thru 0x037047 (0x544a bytes) 000132 1789403234.849967 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000133 1789403234.982558 write 0x0283ea thru 0x036cdb (0xe8f2 bytes) 000135 1789403234.997920 write 0x026f6e thru 0x0274fa (0x58d bytes) 000136 1789403235.007839 mapread 0x009b05 thru 0x015f32 (0xc42e bytes) 000137 1789403235.031561 mapwrite 0x01ce2a thru 0x02837a (0xb551 bytes) 000143 1789403235.107605 read 0x00baf6 thru 0x00eca1 (0x31ac bytes) 000144 1789403235.109173 mapwrite 0x000227 thru 0x00a8e5 (0xa6bf bytes) 000145 1789403235.172406 MIRROR_EXTEND to 2 mirrors 000148 1789403236.005300 trunc from 0x037048 to 0x02b425 (0xbc22 bytes) 000150 1789403236.097989 write 0x014b6a thru 0x01c035 (0x74cc bytes) 000151 1789403236.124867 read 0x0001f6 thru 0x00ffb7 (0xfdc2 bytes) 000152 1789403236.134224 trunc from 0x02b426 to 0x021e42 (0x95e3 bytes) 000153 1789403236.153113 read 0x0124e3 thru 0x018c42 (0x6760 bytes) 000155 1789403236.181601 write 0x016764 thru 0x016d4e (0x5eb bytes) 000160 1789403236.195272 mapwrite 0x03ad16 thru 0x03ffff (0x52ea bytes) 000161 1789403236.277366 MIRROR_RESYNC 2 mirrors 000163 1789403237.024803 trunc from 0x040000 to 0x01af2b (0x250d4 bytes) 000164 1789403237.192401 mapwrite 0x00a678 thru 0x01560b (0xaf94 bytes) 000166 1789403237.280393 trunc from 0x01af2c to 0x005239 (0x15cf2 bytes) 000167 1789403237.301887 read 0x004b37 thru 0x005239 (0x703 bytes) 000172 1789403237.311698 mapread 0x001786 thru 0x005239 (0x3ab4 bytes) 000175 1789403237.319146 write 0x02bf9f thru 0x0368ea (0xa94c bytes) 000176 1789403237.321580 write 0x0306a4 thru 0x03d5c7 (0xcf24 bytes) 000177 1789403237.323443 mapread 0x00eaa3 thru 0x01cd8e (0xe2ec bytes) 000178 1789403237.336895 trunc from 0x03d5c8 to 0x0251f1 (0x183d6 bytes) 000179 1789403237.380096 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000180 1789403237.612519 trunc from 0x0251f2 to 0x03875f (0x1356e bytes) 000181 1789403237.632630 mapwrite 0x01a875 thru 0x028751 (0xdedd bytes) 000183 1789403237.848907 mapwrite 0x023386 thru 0x02c755 (0x93d0 bytes) 000185 1789403237.977146 write 0x00cfc7 thru 0x0158aa (0x88e4 bytes) 000186 1789403237.991155 mapread 0x00525c thru 0x012bf0 (0xd995 bytes) 000189 1789403238.014469 mapwrite 0x018354 thru 0x01d3f2 (0x509f bytes) 000190 1789403238.100370 mapread 0x027a66 thru 0x035136 (0xd6d1 bytes) 000191 1789403238.112803 trunc from 0x038760 to 0x00b941 (0x2ce1e bytes) 000192 1789403238.144348 write 0x01028b thru 0x01dfb3 (0xdd29 bytes) 000193 1789403238.146927 read 0x00137c thru 0x00eb54 (0xd7d9 bytes) 000195 1789403238.156557 trunc from 0x01dfb4 to 0x00dfdf (0xffd4 bytes) 000197 1789403238.174753 mapread 0x0020d9 thru 0x00dfdf (0xbf07 bytes) 000199 1789403238.177134 trunc from 0x00dfe0 to 0x015b3f (0x7b60 bytes) 000200 1789403238.201450 write 0x038813 thru 0x03ffff (0x77ed bytes) 000201 1789403238.209515 write 0x02aaab thru 0x032a4b (0x7fa1 bytes) 000202 1789403238.250388 MIRROR_EXTEND to 2 mirrors 000203 1789403239.268877 trunc from 0x040000 to 0x00286d (0x3d792 bytes) 000204 1789403239.351096 mapread 0x001559 thru 0x00286d (0x1315 bytes) 000206 1789403239.361253 mapwrite 0x00ceb6 thru 0x01bcbc (0xee07 bytes) 000207 1789403239.455728 write 0x038321 thru 0x03ffff (0x7cdf bytes) 000208 1789403239.456859 mapread 0x0196c3 thru 0x0275fb (0xdf39 bytes) 000209 1789403239.469159 trunc from 0x040000 to 0x020d3d (0x1f2c2 bytes) 000212 1789403239.485341 write 0x03288c thru 0x03c8a7 (0xa01c bytes) 000213 1789403239.491174 MIRROR_RESYNC 2 mirrors 000214 1789403240.214253 MIRROR_RESYNC 2 mirrors 000215 1789403240.585158 mapwrite 0x03ec8f thru 0x03ffff (0x1371 bytes) 000216 1789403240.712087 punch from 0x03eccb to 0x03ffff (0x1335 bytes) 000217 1789403240.716604 punch from 0x01fb79 to 0x02d903 (0xdd8b bytes) 000218 1789403240.720614 write 0x01af32 thru 0x021d2b (0x6dfa bytes) 000219 1789403240.732311 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000222 1789403240.878413 MIRROR_EXTEND to 2 mirrors 000223 1789403241.737065 punch from 0x015ea6 to 0x022414 (0xc56f bytes) 000227 1789403241.833390 trunc from 0x040000 to 0x00cbc9 (0x33436 bytes) 000228 1789403241.858261 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000229 1789403241.972486 read 0x007fed thru 0x00cbc9 (0x4bdd bytes) 000230 1789403241.977148 mapwrite 0x0043f8 thru 0x00aea0 (0x6aa9 bytes) 000231 1789403242.045215 trunc from 0x00cbca to 0x012528 (0x595f bytes) 000233 1789403242.077586 write 0x004bce thru 0x00676a (0x1b9d bytes) 000234 1789403242.079176 mapwrite 0x03ad63 thru 0x03ffff (0x529d bytes) 000235 1789403242.166843 mapread 0x019da0 thru 0x01d945 (0x3ba6 bytes) 000237 1789403242.178181 mapwrite 0x022926 thru 0x0292d8 (0x69b3 bytes) 000238 1789403242.272107 read 0x029400 thru 0x02bb1a (0x271b bytes) 000239 1789403242.290229 punch from 0x03b9d4 to 0x03ffff (0x462c bytes) 000240 1789403242.295563 mapwrite 0x034c82 thru 0x03ffff (0xb37e bytes) 000241 1789403242.380225 write 0x010a5b thru 0x0138bf (0x2e65 bytes) 000242 1789403242.381690 mapwrite 0x03db21 thru 0x03ffff (0x24df bytes) 000243 1789403242.436773 trunc from 0x040000 to 0x00ba6d (0x34592 bytes) 000244 1789403242.465576 write 0x018817 thru 0x018f95 (0x77f bytes) 000247 1789403242.474079 MIRROR_EXTEND to 2 mirrors 000248 1789403243.434980 mapread 0x015497 thru 0x018f95 (0x3aff bytes) 000249 1789403243.467751 read 0x002535 thru 0x004fa0 (0x2a6c bytes) 000250 1789403243.472627 write 0x03e07d thru 0x03e18a (0x10e bytes) 000252 1789403243.591812 write 0x023912 thru 0x02ee7c (0xb56b bytes) 000253 1789403243.619564 read 0x00129a thru 0x0066da (0x5441 bytes) 000255 1789403243.644963 mapwrite 0x035b74 thru 0x03d78f (0x7c1c bytes) 000256 1789403243.726845 read 0x02fc88 thru 0x037da4 (0x811d bytes) 000258 1789403243.728539 read 0x02065c thru 0x02a91e (0xa2c3 bytes) 000260 1789403243.730271 write 0x0204c6 thru 0x02da5e (0xd599 bytes) 000261 1789403243.732722 read 0x038400 thru 0x03e18a (0x5d8b bytes) 000262 1789403243.733826 punch from 0x02e5ca to 0x03577e (0x71b5 bytes) 000263 1789403243.744342 mapread 0x0280b1 thru 0x02e01c (0x5f6c bytes) 000264 1789403243.758092 mapwrite 0x019bd8 thru 0x01ac40 (0x1069 bytes) 000266 1789403243.804559 write 0x0324a1 thru 0x03c55d (0xa0bd bytes) 000267 1789403243.806476 write 0x01c9a5 thru 0x01e08e (0x16ea bytes) 000268 1789403243.811478 write 0x0037ce thru 0x0126fd (0xef30 bytes) 000269 1789403243.813828 punch from 0x0324aa to 0x03bc6b (0x97c2 bytes) 000270 1789403243.832544 read 0x00f078 thru 0x01a876 (0xb7ff bytes) 000271 1789403243.833695 mapwrite 0x025226 thru 0x028d1b (0x3af6 bytes) 000272 1789403243.882807 mapread 0x02d8d6 thru 0x03ceeb (0xf616 bytes) 000274 1789403243.891802 trunc from 0x03e18b to 0x0298d8 (0x148b2 bytes) 000276 1789403243.914549 mapread 0x01365e thru 0x0204a1 (0xce44 bytes) 000277 1789403243.915492 mapwrite 0x0176ea thru 0x017c3f (0x556 bytes) 000281 1789403243.954830 mapwrite 0x03b19a thru 0x03ffff (0x4e66 bytes) 000283 1789403244.023497 mapread 0x019288 thru 0x0216d9 (0x8452 bytes) 000284 1789403244.024441 punch from 0x00993d to 0x019697 (0xfd5b bytes) 000285 1789403244.029524 mapwrite 0x016c3e thru 0x024f56 (0xe319 bytes) 000288 1789403244.097420 mapread 0x00f3dc thru 0x0151e3 (0x5e08 bytes) 000290 1789403244.127090 trunc from 0x040000 to 0x016a37 (0x295c8 bytes) 000291 1789403244.148617 mapread 0x008769 thru 0x013d64 (0xb5fc bytes) 000293 1789403244.174248 mapread 0x0024e9 thru 0x01207a (0xfb92 bytes) 000294 1789403244.175242 read 0x0052e6 thru 0x006703 (0x141e bytes) 000295 1789403244.176274 write 0x020b66 thru 0x0225b2 (0x1a4d bytes) 000296 1789403244.177634 mapwrite 0x0015ef thru 0x00a798 (0x91aa bytes) 000297 1789403244.218151 write 0x02c284 thru 0x02d824 (0x15a1 bytes) 000301 1789403244.220545 mapwrite 0x0393d5 thru 0x03ffff (0x6c2b bytes) 000302 1789403244.291600 read 0x02ce81 thru 0x034952 (0x7ad2 bytes) 000303 1789403244.292372 read 0x03863d thru 0x03ffff (0x79c3 bytes) 000304 1789403244.301398 mapwrite 0x01779e thru 0x01d3ff (0x5c62 bytes) 000305 1789403244.362516 punch from 0x00b94d to 0x0164ad (0xab61 bytes) 000306 1789403244.366597 read 0x02861f thru 0x036245 (0xdc27 bytes) 000307 1789403244.367296 punch from 0x00fcc2 to 0x01e49d (0xe7dc bytes) 000310 1789403244.372823 trunc from 0x040000 to 0x00cce4 (0x3331b bytes) 000311 1789403244.385186 write 0x01d3a5 thru 0x01d78c (0x3e8 bytes) 000312 1789403244.386514 mapread 0x01d728 thru 0x01d78c (0x65 bytes) 000313 1789403244.387566 punch from 0x011271 to 0x01979f (0x852f bytes) 000314 1789403244.395606 write 0x008d9d thru 0x013aea (0xad4e bytes) 000315 1789403244.405228 read 0x017934 thru 0x01d78c (0x5e59 bytes) 000316 1789403244.412844 punch from 0x00fc76 to 0x01a93f (0xacca bytes) 000317 1789403244.423486 write 0x00f1ff thru 0x014fdf (0x5de1 bytes) 000318 1789403244.431193 read 0x008b38 thru 0x012b18 (0x9fe1 bytes) 000320 1789403244.432597 write 0x039540 thru 0x03cda4 (0x3865 bytes) 000321 1789403244.433569 punch from 0x01c168 to 0x01f36c (0x3205 bytes) 000322 1789403244.443931 mapread 0x00a41f thru 0x0118e8 (0x74ca bytes) 000323 1789403244.444833 trunc from 0x03cda5 to 0x01ecfd (0x1e0a7 bytes) 000324 1789403244.462011 mapwrite 0x01e461 thru 0x01ee2c (0x9cc bytes) 000325 1789403244.573054 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000326 1789403244.726394 read 0x014436 thru 0x017b68 (0x3733 bytes) 000328 1789403244.733797 mapwrite 0x011fc7 thru 0x0194a9 (0x74e3 bytes) 000329 1789403244.788549 mapread 0x00764b thru 0x00d3ee (0x5da4 bytes) 000330 1789403244.795588 punch from 0x00f25c to 0x016b19 (0x78be bytes) 000331 1789403244.800309 write 0x021564 thru 0x02287a (0x1317 bytes) 000332 1789403244.801453 trunc from 0x02287b to 0x006187 (0x1c6f3 bytes) 000333 1789403244.816945 mapwrite 0x01ec47 thru 0x01fc97 (0x1051 bytes) 000335 1789403244.887972 read 0x01a619 thru 0x01cfd8 (0x29c0 bytes) 000336 1789403244.889174 mapwrite 0x0165fc thru 0x019648 (0x304d bytes) 000340 1789403244.928190 punch from 0x00e7b2 to 0x011fed (0x383c bytes) 000342 1789403244.932310 read 0x0152ae thru 0x01fc97 (0xa9ea bytes) 000343 1789403244.932849 trunc from 0x01fc98 to 0x03bc70 (0x1bfd9 bytes) 000348 1789403244.966995 trunc from 0x03bc71 to 0x00f0c5 (0x2cbab bytes) 000349 1789403244.991367 trunc from 0x00f0c6 to 0x026484 (0x173bf bytes) 000350 1789403245.024935 mapread 0x0244ef thru 0x026484 (0x1f96 bytes) 000351 1789403245.040530 mapwrite 0x0391ef thru 0x03ffff (0x6e11 bytes) 000353 1789403245.171495 MIRROR_EXTEND to 2 mirrors 000355 1789403246.049216 read 0x000047 thru 0x00b303 (0xb2bd bytes) 000356 1789403246.062232 MIRROR_EXTEND to 3 mirrors 000357 1789403247.019394 punch from 0x006f27 to 0x00e3dc (0x74b6 bytes) 000359 1789403247.070051 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2f99f2b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2f99f2b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000360 1789403247.312474 trunc from 0x040000 to 0x0063ff (0x39c00 bytes) 000363 1789403247.343553 write 0x0135f6 thru 0x0143f4 (0xdff bytes) 000364 1789403247.345313 read 0x01193a thru 0x0143f4 (0x2abb bytes) 000365 1789403247.351885 read 0x00ba2c thru 0x0143f4 (0x89c9 bytes) 000366 1789403247.361443 write 0x008e9b thru 0x016313 (0xd479 bytes) 000367 1789403247.366068 mapwrite 0x016649 thru 0x01c292 (0x5c4a bytes) 000368 1789403247.472538 write 0x01128d thru 0x012a80 (0x17f4 bytes) 000369 1789403247.473604 mapwrite 0x03bbd9 thru 0x03ffff (0x4427 bytes) 000370 1789403247.600974 mapread 0x015688 thru 0x016e86 (0x17ff bytes) 000371 1789403247.611482 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000373 1789403247.817623 mapwrite 0x002dde thru 0x012a20 (0xfc43 bytes) 000374 1789403247.944463 punch from 0x0218c1 to 0x02e090 (0xc7d0 bytes) 000377 1789403247.954557 trunc from 0x040000 to 0x02475e (0x1b8a1 bytes) 000379 1789403247.971797 mapwrite 0x013431 thru 0x01d5e4 (0xa1b4 bytes) 000380 1789403248.062632 read 0x01e597 thru 0x02475e (0x61c8 bytes) 000382 1789403248.072478 mapread 0x01b548 thru 0x0225a8 (0x7061 bytes) 000383 1789403248.074011 mapread 0x0224fd thru 0x02475e (0x2262 bytes) 000384 1789403248.076242 punch from 0x01b3fa to 0x0220de (0x6ce5 bytes) 000385 1789403248.080716 trunc from 0x02475f to 0x022489 (0x22d5 bytes) 000387 1789403248.092763 mapwrite 0x03b0d8 thru 0x03ffff (0x4f28 bytes) 000388 1789403248.149935 MIRROR_EXTEND to 2 mirrors 000390 1789403249.068288 mapread 0x01af5a thru 0x027218 (0xc2bf bytes) 000391 1789403249.114563 write 0x03ba43 thru 0x03ffff (0x45bd bytes) 000392 1789403249.255601 mapread 0x01df86 thru 0x02d9d3 (0xfa4e bytes) 000393 1789403249.271118 punch from 0x01ab23 to 0x0214cc (0x69aa bytes) 000394 1789403249.287463 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xfa8b6f62 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2220cdb3 Mirror 3: 0x2220cdb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000395 1789403250.216576 punch from 0x00721b to 0x00d445 (0x622b bytes) 000398 1789403250.291537 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2220cdb3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2220cdb3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000399 1789403250.518483 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000400 1789403250.624478 write 0x00e113 thru 0x012055 (0x3f43 bytes) 000401 1789403250.635960 trunc from 0x040000 to 0x03ce4c (0x31b3 bytes) 000402 1789403250.673708 trunc from 0x03ce4d to 0x018b07 (0x24345 bytes) 000403 1789403250.695069 punch from 0x0101d4 to 0x01429e (0x40cb bytes) 000404 1789403250.706599 read 0x006ad1 thru 0x00c5b4 (0x5ae4 bytes) 000405 1789403250.714614 punch from 0x00fdb8 to 0x014023 (0x426c bytes) 000407 1789403250.719504 trunc from 0x018b08 to 0x03b372 (0x2286b bytes) 000408 1789403250.735836 read 0x0166d7 thru 0x017be5 (0x150f bytes) 000409 1789403250.745482 trunc from 0x03b373 to 0x02d6b3 (0xdcbf bytes) 000410 1789403250.760018 read 0x00d2c7 thru 0x01ab84 (0xd8be bytes) 000411 1789403250.761199 write 0x0327b8 thru 0x03e027 (0xb870 bytes) 000414 1789403250.765192 mapread 0x007077 thru 0x008522 (0x14ac bytes) 000415 1789403250.766199 trunc from 0x03e028 to 0x01000a (0x2e01d bytes) 000416 1789403250.782952 trunc from 0x01000b to 0x036310 (0x26306 bytes) 000418 1789403250.805180 write 0x01a614 thru 0x02a5d8 (0xffc5 bytes) 000419 1789403250.824926 mapread 0x01e292 thru 0x023f05 (0x5c74 bytes) 000421 1789403250.827103 read 0x017ca5 thru 0x01859e (0x8fa bytes) 000422 1789403250.827995 read 0x0045ba thru 0x010a8c (0xc4d3 bytes) 000423 1789403250.832139 MIRROR_EXTEND to 2 mirrors 000424 1789403251.841287 trunc from 0x036311 to 0x01f3fc (0x16f14 bytes) 000425 1789403251.950394 mapread 0x00236a thru 0x00c839 (0xa4d0 bytes) 000426 1789403251.975116 mapwrite 0x02362e thru 0x02e431 (0xae04 bytes) 000428 1789403252.088565 mapwrite 0x0264d9 thru 0x02aa24 (0x454c bytes) 000430 1789403252.132279 mapwrite 0x01e56c thru 0x02dada (0xf56f bytes) 000431 1789403252.197283 punch from 0x017aa4 to 0x01ae72 (0x33cf bytes) 000433 1789403252.202078 punch from 0x018ef8 to 0x01bfff (0x3108 bytes) 000434 1789403252.207566 trunc from 0x02e432 to 0x02f074 (0xc43 bytes) 000435 1789403252.230415 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000436 1789403252.382954 trunc from 0x02f075 to 0x02d053 (0x2021 bytes) 000438 1789403252.415770 write 0x00e3fd thru 0x01067c (0x2280 bytes) 000440 1789403252.429949 mapwrite 0x00a5ef thru 0x01a3b4 (0xfdc6 bytes) 000444 1789403252.529027 mapwrite 0x000a29 thru 0x000fc8 (0x5a0 bytes) 000446 1789403252.577079 trunc from 0x02d054 to 0x008ae0 (0x24573 bytes) 000447 1789403252.592636 punch from 0x001110 to 0x007890 (0x6781 bytes) 000448 1789403252.603987 MIRROR_EXTEND to 2 mirrors 000449 1789403253.486404 MIRROR_EXTEND to 3 mirrors 000450 1789403254.183738 trunc from 0x008ae1 to 0x01162d (0x8b4d bytes) 000451 1789403254.255165 read 0x000c90 thru 0x010923 (0xfc94 bytes) 000453 1789403254.266279 trunc from 0x01162e to 0x004771 (0xcebc bytes) 000455 1789403254.290513 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4772): Mirror 2: 0xd2adfef5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4772): Mirror 2: 0xd2adfef5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4772: skipped 000457 1789403254.505481 read 0x003c1b thru 0x004771 (0xb57 bytes) 000458 1789403254.511371 mapread 0x0010e7 thru 0x004771 (0x368b bytes) 000461 1789403254.522459 punch from 0x001a9b to 0x004771 (0x2cd7 bytes) 000462 1789403254.526901 mapwrite 0x035474 thru 0x0361f8 (0xd85 bytes) 000463 1789403254.592467 trunc from 0x0361f9 to 0x03c2be (0x60c6 bytes) 000464 1789403254.620973 mapwrite 0x033f3f thru 0x0366de (0x27a0 bytes) 000465 1789403254.718096 mapwrite 0x003752 thru 0x010036 (0xc8e5 bytes) 000466 1789403254.857710 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000467 1789403254.982409 punch from 0x00ae83 to 0x00d973 (0x2af1 bytes) 000468 1789403254.988593 read 0x01db6c thru 0x02b726 (0xdbbb bytes) 000469 1789403255.000045 write 0x026399 thru 0x0264f8 (0x160 bytes) 000472 1789403255.002583 mapwrite 0x010be0 thru 0x018305 (0x7726 bytes) 000473 1789403255.080110 mapread 0x00448c thru 0x012a67 (0xe5dc bytes) 000475 1789403255.093252 punch from 0x0333b5 to 0x03b5b0 (0x81fc bytes) 000476 1789403255.099518 punch from 0x03950f to 0x03c2be (0x2db0 bytes) 000479 1789403255.107043 write 0x013ad2 thru 0x01bc1a (0x8149 bytes) 000481 1789403255.114139 trunc from 0x03c2bf to 0x01ed39 (0x1d585 bytes) 000483 1789403255.129807 trunc from 0x01ed3a to 0x002d78 (0x1bfc1 bytes) 000485 1789403255.155056 read 0x001ca2 thru 0x002d78 (0x10d7 bytes) 000489 1789403255.158431 write 0x0370fe thru 0x03ffff (0x8f02 bytes) 000490 1789403255.160367 mapread 0x02f620 thru 0x032f4b (0x392c bytes) 000491 1789403255.176379 MIRROR_EXTEND to 2 mirrors 000492 1789403255.869211 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000493 1789403255.994743 mapwrite 0x00eb6f thru 0x014df1 (0x6283 bytes) 000494 1789403256.071521 read 0x004ace thru 0x011aa3 (0xcfd6 bytes) 000495 1789403256.078020 trunc from 0x040000 to 0x001da3 (0x3e25c bytes) 000498 1789403256.118589 mapwrite 0x0240b1 thru 0x0303dd (0xc32d bytes) 000499 1789403256.189619 mapwrite 0x00a87b thru 0x016b98 (0xc31e bytes) 000502 1789403256.246953 mapwrite 0x039cf6 thru 0x03a84e (0xb59 bytes) 000504 1789403256.313057 punch from 0x005718 to 0x00caba (0x73a3 bytes) 000505 1789403256.319968 write 0x01fe76 thru 0x02b33b (0xb4c6 bytes) 000507 1789403256.324474 mapread 0x01cdbd thru 0x023877 (0x6abb bytes) 000508 1789403256.330468 mapwrite 0x018b6b thru 0x025a13 (0xcea9 bytes) 000510 1789403256.434909 mapread 0x0386aa thru 0x03a84e (0x21a5 bytes) 000514 1789403256.440854 mapread 0x00966b thru 0x014fc1 (0xb957 bytes) 000517 1789403256.454107 punch from 0x014e51 to 0x01bb3d (0x6ced bytes) 000518 1789403256.459882 trunc from 0x03a84f to 0x0213b5 (0x19499 bytes) 000519 1789403256.478141 mapread 0x01b0a7 thru 0x01d792 (0x26ec bytes) 000521 1789403256.488199 mapread 0x01addf thru 0x0213b5 (0x65d7 bytes) 000523 1789403256.489633 mapread 0x001496 thru 0x00ced5 (0xba40 bytes) 000525 1789403256.496457 read 0x002315 thru 0x002feb (0xcd7 bytes) 000526 1789403256.497230 mapread 0x020b5a thru 0x0213b5 (0x85c bytes) 000527 1789403256.498311 mapread 0x020e0d thru 0x0213b5 (0x5a9 bytes) 000528 1789403256.503037 MIRROR_EXTEND to 2 mirrors 000531 1789403257.322886 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000532 1789403257.424056 punch from 0x005f42 to 0x00d5eb (0x76aa bytes) 000535 1789403257.436534 mapread 0x00d621 thru 0x01697c (0x935c bytes) 000537 1789403257.456205 mapread 0x010f03 thru 0x0141ed (0x32eb bytes) 000538 1789403257.457064 punch from 0x01b9ec to 0x01ee10 (0x3425 bytes) 000539 1789403257.462342 read 0x014a14 thru 0x017ddf (0x33cc bytes) 000540 1789403257.463024 read 0x00dc0c thru 0x016acf (0x8ec4 bytes) 000541 1789403257.464147 mapwrite 0x00c253 thru 0x014943 (0x86f1 bytes) 000542 1789403257.506731 trunc from 0x0213b6 to 0x01a423 (0x6f92 bytes) 000543 1789403257.520457 mapwrite 0x010f3c thru 0x01aa54 (0x9b19 bytes) 000544 1789403257.586299 MIRROR_EXTEND to 2 mirrors 000545 1789403258.441763 mapwrite 0x01b037 thru 0x01f49f (0x4469 bytes) 000546 1789403258.614027 write 0x02db94 thru 0x038aeb (0xaf58 bytes) 000547 1789403258.616192 trunc from 0x038aec to 0x03a237 (0x174c bytes) 000549 1789403258.639121 read 0x031ab9 thru 0x036229 (0x4771 bytes) 000551 1789403258.641020 write 0x03cc8f thru 0x03ffff (0x3371 bytes) 000552 1789403258.642267 mapwrite 0x034e23 thru 0x03ffff (0xb1dd bytes) 000554 1789403258.689948 mapread 0x02078d thru 0x02cc69 (0xc4dd bytes) 000555 1789403258.703596 read 0x01a36b thru 0x01f999 (0x562f bytes) 000556 1789403258.704192 mapwrite 0x018f58 thru 0x01d28a (0x4333 bytes) 000558 1789403258.749814 read 0x006098 thru 0x010d94 (0xacfd bytes) 000560 1789403258.759726 punch from 0x01463a to 0x01ff22 (0xb8e9 bytes) 000561 1789403258.763804 write 0x0201c8 thru 0x0263ca (0x6203 bytes) 000565 1789403258.767563 mapwrite 0x0395d2 thru 0x03ffff (0x6a2e bytes) 000566 1789403258.838513 mapwrite 0x022f71 thru 0x02f89a (0xc92a bytes) 000567 1789403258.933234 write 0x03a5a2 thru 0x03ffff (0x5a5e bytes) 000568 1789403258.934319 read 0x0087b2 thru 0x0096c4 (0xf13 bytes) 000569 1789403258.934890 write 0x03b4aa thru 0x03ffff (0x4b56 bytes) 000570 1789403258.937619 punch from 0x02db30 to 0x03d69e (0xfb6f bytes) 000571 1789403258.956231 mapread 0x0164d6 thru 0x02296c (0xc497 bytes) 000572 1789403258.975452 mapwrite 0x0150cb thru 0x0240eb (0xf021 bytes) 000573 1789403259.053794 mapread 0x03fb3b thru 0x03ffff (0x4c5 bytes) 000574 1789403259.055755 mapread 0x030a24 thru 0x0392ab (0x8888 bytes) 000575 1789403259.058978 punch from 0x02012f to 0x02d761 (0xd633 bytes) 000576 1789403259.067305 mapwrite 0x0025ab thru 0x0090c4 (0x6b1a bytes) 000577 1789403259.141481 write 0x00d0a3 thru 0x00fac7 (0x2a25 bytes) 000578 1789403259.142573 read 0x02d6c1 thru 0x0360dd (0x8a1d bytes) 000579 1789403259.155520 trunc from 0x040000 to 0x00221f (0x3dde0 bytes) 000581 1789403259.174009 mapwrite 0x032349 thru 0x037cda (0x5992 bytes) 000582 1789403259.256701 mapread 0x01308d thru 0x013bbf (0xb33 bytes) 000583 1789403259.262786 mapread 0x022d41 thru 0x02951a (0x67da bytes) 000586 1789403259.306936 trunc from 0x037cdb to 0x03f840 (0x7b66 bytes) 000589 1789403259.319935 write 0x02c84c thru 0x02e053 (0x1808 bytes) 000590 1789403259.336622 trunc from 0x03f841 to 0x039f12 (0x592e bytes) 000592 1789403259.350476 read 0x01f75a thru 0x01fe83 (0x72a bytes) 000593 1789403259.360334 punch from 0x00db43 to 0x01d18f (0xf64d bytes) 000594 1789403259.368844 read 0x01b302 thru 0x02520a (0x9f09 bytes) 000595 1789403259.376541 trunc from 0x039f13 to 0x031509 (0x8a09 bytes) 000596 1789403259.388413 write 0x0222f1 thru 0x02773f (0x544f bytes) 000597 1789403259.392150 MIRROR_RESYNC 2 mirrors 000598 1789403260.166236 MIRROR_EXTEND to 3 mirrors 000599 1789403260.923089 read 0x018868 thru 0x01bac6 (0x325f bytes) 000600 1789403260.928022 punch from 0x025735 to 0x031509 (0xbdd5 bytes) 000601 1789403260.989207 read 0x000f8e thru 0x00fa5f (0xead2 bytes) 000604 1789403261.000271 write 0x011cdd thru 0x0219fb (0xfd1f bytes) 000605 1789403261.002513 read 0x00074e thru 0x005657 (0x4f0a bytes) 000606 1789403261.003364 punch from 0x009073 to 0x0132c6 (0xa254 bytes) 000609 1789403261.015060 read 0x026c56 thru 0x031509 (0xa8b4 bytes) 000610 1789403261.019090 MIRROR_RESYNC 3 mirrors 000611 1789403261.748973 trunc from 0x03150a to 0x03dbf6 (0xc6ed bytes) 000612 1789403261.858836 mapwrite 0x00e69c thru 0x0180c0 (0x9a25 bytes) 000613 1789403261.944353 trunc from 0x03dbf7 to 0x026013 (0x17be3 bytes) 000615 1789403261.962522 read 0x0003c5 thru 0x00c72e (0xc36a bytes) 000616 1789403261.970232 punch from 0x01ba19 to 0x026013 (0xa5fb bytes) 000617 1789403261.975801 punch from 0x00434a to 0x012f13 (0xebca bytes) 000618 1789403261.985265 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26014): Mirror 3: 0xcc0c67a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26014): Mirror 3: 0xcc0c67a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26014: skipped 000620 1789403262.316303 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000621 1789403262.429412 write 0x02e8a2 thru 0x0371b9 (0x8918 bytes) 000623 1789403262.431005 trunc from 0x0371ba to 0x03bf07 (0x4d4e bytes) 000624 1789403262.450471 read 0x010208 thru 0x011a9e (0x1897 bytes) 000625 1789403262.461770 trunc from 0x03bf08 to 0x004363 (0x37ba4 bytes) 000626 1789403262.485447 mapwrite 0x02f16a thru 0x030671 (0x1508 bytes) 000627 1789403262.637138 read 0x02337a thru 0x02ae27 (0x7aae bytes) 000630 1789403262.656483 punch from 0x00d757 to 0x014fd6 (0x7880 bytes) 000631 1789403262.684543 MIRROR_EXTEND to 2 mirrors 000633 1789403263.505479 mapread 0x02548c thru 0x02a7df (0x5354 bytes) 000635 1789403263.525220 trunc from 0x030672 to 0x007b26 (0x28b4b bytes) 000636 1789403263.634298 mapwrite 0x0363da thru 0x03ffff (0x9c26 bytes) 000638 1789403263.750415 write 0x009066 thru 0x011f4c (0x8ee7 bytes) 000639 1789403263.771188 mapwrite 0x03f653 thru 0x03ffff (0x9ad bytes) 000640 1789403263.831134 read 0x03af06 thru 0x03ffff (0x50fa bytes) 000641 1789403263.831764 punch from 0x01f424 to 0x02d3e2 (0xdfbf bytes) 000643 1789403263.838019 punch from 0x02392b to 0x02cc64 (0x933a bytes) 000644 1789403263.846045 punch from 0x023c7c to 0x0246c1 (0xa46 bytes) 000646 1789403263.852947 trunc from 0x040000 to 0x016cc4 (0x2933b bytes) 000647 1789403263.872770 write 0x02c384 thru 0x033ed2 (0x7b4f bytes) 000648 1789403263.876008 write 0x00ab10 thru 0x01414d (0x963e bytes) 000649 1789403263.879215 read 0x000db0 thru 0x00d1cb (0xc41c bytes) 000650 1789403263.893056 write 0x034562 thru 0x03ffff (0xba9e bytes) 000651 1789403263.895754 punch from 0x035c2f to 0x037c11 (0x1fe3 bytes) 000653 1789403263.918179 trunc from 0x040000 to 0x0132c9 (0x2cd36 bytes) 000654 1789403263.942629 write 0x009933 thru 0x015bd4 (0xc2a2 bytes) 000657 1789403263.947200 mapwrite 0x03d81a thru 0x03ffff (0x27e6 bytes) 000658 1789403264.046510 trunc from 0x040000 to 0x029758 (0x168a7 bytes) 000659 1789403264.071060 punch from 0x01d400 to 0x01f076 (0x1c77 bytes) 000661 1789403264.077730 read 0x0236d7 thru 0x029758 (0x6082 bytes) 000666 1789403264.094078 read 0x000d46 thru 0x0055eb (0x48a6 bytes) 000668 1789403264.096992 read 0x002fe3 thru 0x005f45 (0x2f63 bytes) 000670 1789403264.105189 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000671 1789403264.247326 read 0x023de5 thru 0x0266c7 (0x28e3 bytes) 000674 1789403264.255952 punch from 0x016911 to 0x0179a1 (0x1091 bytes) 000676 1789403264.265289 read 0x003ab3 thru 0x00926d (0x57bb bytes) 000677 1789403264.279923 trunc from 0x029759 to 0x03575b (0xc003 bytes) 000679 1789403264.308192 MIRROR_EXTEND to 2 mirrors 000680 1789403265.425938 read 0x012e22 thru 0x015064 (0x2243 bytes) 000683 1789403265.442927 mapwrite 0x0074d6 thru 0x012ab5 (0xb5e0 bytes) 000684 1789403265.842869 MIRROR_RESYNC 2 mirrors 000685 1789403267.005325 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) 000686 1789403267.108853 write 0x00465e thru 0x0092ba (0x4c5d bytes) 000687 1789403267.128549 mapwrite 0x0260a1 thru 0x034ce1 (0xec41 bytes) 000688 1789403267.198178 punch from 0x0317d0 to 0x03575b (0x3f8c bytes) 000689 1789403267.202726 write 0x03beab thru 0x03ffff (0x4155 bytes) 000690 1789403267.203831 punch from 0x011986 to 0x01586a (0x3ee5 bytes) 000691 1789403267.214966 mapread 0x020513 thru 0x02cca3 (0xc791 bytes) 000694 1789403267.225520 punch from 0x037c1b to 0x03ffff (0x83e5 bytes) 000695 1789403267.229623 mapwrite 0x02e0b1 thru 0x0300ca (0x201a bytes) 000697 1789403267.280524 MIRROR_EXTEND to 2 mirrors 000698 1789403268.045149 write 0x010c96 thru 0x01f143 (0xe4ae bytes) 000700 1789403268.136598 read 0x030620 thru 0x031867 (0x1248 bytes) 000701 1789403268.152325 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x9d3cbf92 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe911a195 Mirror 3: 0xe911a195 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000703 1789403269.474241 write 0x0121d2 thru 0x020e21 (0xec50 bytes) 000704 1789403269.582183 mapwrite 0x000828 thru 0x003727 (0x2f00 bytes) 000705 1789403269.699080 MIRROR_RESYNC 3 mirrors 000706 1789403270.925055 mapwrite 0x015bde thru 0x01640a (0x82d bytes) 000708 1789403271.068934 read 0x010033 thru 0x01f608 (0xf5d6 bytes) 000709 1789403271.077293 trunc from 0x040000 to 0x0220e2 (0x1df1d bytes) 000711 1789403271.105924 read 0x01ebe9 thru 0x0220e2 (0x34fa bytes) 000714 1789403271.123770 trunc from 0x0220e3 to 0x037023 (0x14f41 bytes) 000715 1789403271.144170 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37024): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37024): Mirror 2: 0xc337537e (stale) Mirror 3: 0xc337537e (stale) CRC-32 checksum value for chunk [0, 0x37024): Mirror 1: 0xc76614b3 Mirror 4: 0xc76614b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37024: skipped 000716 1789403272.067274 punch from 0x02a6e7 to 0x02ea40 (0x435a bytes) 000718 1789403272.145820 trunc from 0x037024 to 0x036fdc (0x47 bytes) 000719 1789403272.188225 read 0x01ac52 thru 0x028c49 (0xdff8 bytes) 000720 1789403272.194989 mapwrite 0x01c960 thru 0x026076 (0x9717 bytes) 000722 1789403272.265774 trunc from 0x036fdd to 0x00e094 (0x28f48 bytes) 000723 1789403272.277546 trunc from 0x00e095 to 0x02bb47 (0x1dab3 bytes) 000725 1789403272.292623 mapwrite 0x013740 thru 0x01a937 (0x71f8 bytes) 000726 1789403272.368387 punch from 0x01c24d to 0x023b40 (0x78f4 bytes) 000727 1789403272.375325 trunc from 0x02bb48 to 0x03bd89 (0x10242 bytes) 000728 1789403272.401500 MIRROR_RESYNC 4 mirrors 000730 1789403273.718473 MIRROR_EXTEND to 5 mirrors 000731 1789403274.364518 trunc from 0x03bd8a to 0x015f5b (0x25e2e bytes) 000732 1789403274.457478 trunc from 0x015f5c to 0x0166ca (0x76f bytes) 000734 1789403274.479682 mapwrite 0x01783b thru 0x025c0f (0xe3d5 bytes) 000738 1789403274.586776 mapread 0x001bd6 thru 0x00b532 (0x995d bytes) 000739 1789403274.594459 mapread 0x0204dd thru 0x025c0f (0x5733 bytes) 000740 1789403274.595429 trunc from 0x025c10 to 0x01c270 (0x999f bytes) 000741 1789403274.614267 write 0x0368bc thru 0x03ffff (0x9744 bytes) 000743 1789403274.620871 MIRROR_RESYNC 5 mirrors 000744 1789403276.891616 write 0x025e87 thru 0x0337d7 (0xd951 bytes) 000746 1789403276.972462 mapwrite 0x036985 thru 0x03ffff (0x967b bytes) 000748 1789403277.085862 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7badb58b (stale) Mirror 3: 0x7badb58b (stale) Mirror 4: 0x7badb58b (stale) Mirror 5: 0x7badb58b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd0da1199 Mirror 6: 0xd0da1199 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000751 1789403278.304157 trunc from 0x040000 to 0x031220 (0xeddf bytes) 000752 1789403278.410155 mapread 0x00b36b thru 0x00e4fc (0x3192 bytes) 000754 1789403278.428818 mapread 0x024454 thru 0x030079 (0xbc26 bytes) 000755 1789403278.449261 trunc from 0x031221 to 0x019d64 (0x174bc bytes) 000756 1789403278.478059 mapwrite 0x007d6d thru 0x0165ba (0xe84e bytes) 000757 1789403278.591738 trunc from 0x019d65 to 0x0246fb (0xa997 bytes) 000759 1789403278.638940 mapwrite 0x0017db thru 0x00f271 (0xda97 bytes) 000761 1789403278.765581 trunc from 0x0246fc to 0x0019bc (0x22d3f bytes) 000763 1789403278.787553 trunc from 0x0019bd to 0x0342dd (0x32921 bytes) 000764 1789403278.811920 punch from 0x0338cc to 0x0342dd (0xa12 bytes) 000765 1789403278.817817 trunc from 0x0342de to 0x03f1d3 (0xaef6 bytes) 000767 1789403278.842156 write 0x02d31e thru 0x034a40 (0x7723 bytes) 000768 1789403278.853742 read 0x022daa thru 0x029b23 (0x6d7a bytes) 000770 1789403278.862001 trunc from 0x03f1d4 to 0x035c19 (0x95ba bytes) 000771 1789403278.899785 mapread 0x02281e thru 0x0278da (0x50bd bytes) 000773 1789403278.905966 mapread 0x02bc4d thru 0x035c19 (0x9fcd bytes) 000775 1789403278.916021 punch from 0x01d9ad to 0x0284e9 (0xab3d bytes) 000776 1789403278.938359 punch from 0x003d4e to 0x009ddf (0x6092 bytes) 000777 1789403278.943803 mapwrite 0x039381 thru 0x03ffff (0x6c7f bytes) 000778 1789403279.031217 read 0x03fbb3 thru 0x03ffff (0x44d bytes) 000779 1789403279.032478 mapwrite 0x005dd2 thru 0x013ced (0xdf1c bytes) 000781 1789403279.099734 write 0x02fc1a thru 0x03d74f (0xdb36 bytes) 000782 1789403279.100835 punch from 0x033a79 to 0x03e9f9 (0xaf81 bytes) 000783 1789403279.113145 mapwrite 0x00309f thru 0x00880b (0x576d bytes) 000785 1789403279.158124 mapwrite 0x02014d thru 0x026814 (0x66c8 bytes) 000787 1789403279.207398 punch from 0x00486b to 0x005973 (0x1109 bytes) 000789 1789403279.213189 mapwrite 0x006b2d thru 0x007f24 (0x13f8 bytes) 000790 1789403279.250823 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7badb58b (stale) Mirror 3: 0x7badb58b (stale) Mirror 4: 0x7badb58b (stale) Mirror 5: 0x7badb58b (stale) Mirror 6: 0xd0da1199 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x64822148 Mirror 7: 0x64822148 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000791 1789403280.168162 trunc from 0x040000 to 0x032786 (0xd879 bytes) 000792 1789403280.269062 read 0x004331 thru 0x00591b (0x15eb bytes) 000793 1789403280.274356 mapwrite 0x01fc2d thru 0x0210c5 (0x1499 bytes) 000794 1789403280.327045 trunc from 0x032787 to 0x013053 (0x1f733 bytes) 000795 1789403280.360519 trunc from 0x013054 to 0x01df6a (0xaf17 bytes) 000796 1789403280.377458 mapread 0x00717e thru 0x012fe7 (0xbe6a bytes) 000799 1789403280.397462 mapwrite 0x0132bb thru 0x013839 (0x57f bytes) 000800 1789403280.448135 mapread 0x0074f5 thru 0x009081 (0x1b8d bytes) 000801 1789403280.450335 read 0x0180d6 thru 0x01c686 (0x45b1 bytes) 000802 1789403280.461056 trunc from 0x01df6b to 0x013c53 (0xa317 bytes) 000804 1789403280.490353 mapwrite 0x0245ec thru 0x027a9c (0x34b1 bytes) 000805 1789403280.644083 punch from 0x01a997 to 0x023aa9 (0x9113 bytes) 000808 1789403280.658157 mapwrite 0x006e04 thru 0x00da30 (0x6c2d bytes) 000809 1789403280.748693 mapread 0x006f65 thru 0x0094b3 (0x254f bytes) 000810 1789403280.751578 mapread 0x014018 thru 0x01e80a (0xa7f3 bytes) 000811 1789403280.862731 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27a9d): Mirror 2: 0xb2ea6de8 (stale) Mirror 4: 0xb2ea6de8 (stale) Mirror 5: 0xb2ea6de8 (stale) Mirror 6: 0x9aa350f1 (stale) Mirror 7: 0x8bf0e872 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a9d): Mirror 2: 0xb2ea6de8 (stale) Mirror 4: 0xb2ea6de8 (stale) Mirror 5: 0xb2ea6de8 (stale) Mirror 6: 0x9aa350f1 (stale) Mirror 7: 0x8bf0e872 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a9d: skipped 000812 1789403281.239423 trunc from 0x027a9d to 0x035fd3 (0xe537 bytes) 000813 1789403281.257469 read 0x00e2da thru 0x01dfb0 (0xfcd7 bytes) 000816 1789403281.267024 mapwrite 0x02b051 thru 0x03a21b (0xf1cb bytes) 000817 1789403281.359079 MIRROR_RESYNC 6 mirrors 000818 1789403284.540736 mapwrite 0x01c713 thru 0x029702 (0xcff0 bytes) 000820 1789403284.737790 read 0x034a2c thru 0x03a21b (0x57f0 bytes) 000823 1789403284.747060 mapwrite 0x0121ce thru 0x01be90 (0x9cc3 bytes) 000824 1789403284.814378 punch from 0x0090e3 to 0x00af81 (0x1e9f bytes) 000825 1789403284.841689 mapwrite 0x0394b2 thru 0x03ffff (0x6b4e bytes) 000827 1789403284.963624 trunc from 0x040000 to 0x00d426 (0x32bd9 bytes) 000828 1789403284.987582 read 0x008d0c thru 0x00bfb4 (0x32a9 bytes) 000829 1789403284.995616 read 0x0093ab thru 0x00d426 (0x407c bytes) 000830 1789403284.996901 punch from 0x003100 to 0x003689 (0x58a bytes) 000831 1789403285.000985 trunc from 0x00d427 to 0x017330 (0x9f0a bytes) 000832 1789403285.024441 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x17331): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17331): Mirror 2: 0xd003a5f1 (stale) Mirror 4: 0xd003a5f1 (stale) Mirror 5: 0xd003a5f1 (stale) Mirror 6: 0xd003a5f1 (stale) Mirror 7: 0xd003a5f1 (stale) CRC-32 checksum value for chunk [0, 0x17331): Mirror 1: 0xd45f85e Mirror 8: 0xd45f85e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17331: skipped 000833 1789403286.187476 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17331): Mirror 4: 0xd003a5f1 (stale) Mirror 5: 0xd003a5f1 (stale) Mirror 6: 0xd003a5f1 (stale) Mirror 7: 0xd003a5f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17331): Mirror 4: 0xd003a5f1 (stale) Mirror 5: 0xd003a5f1 (stale) Mirror 6: 0xd003a5f1 (stale) Mirror 7: 0xd003a5f1 (stale) CRC-32 checksum value for chunk [0, 0x17331): Mirror 1: 0xd45f85e Mirror 8: 0xd45f85e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17331: skipped 000835 1789403286.720987 write 0x02414a thru 0x02df3b (0x9df2 bytes) 000836 1789403286.804461 mapwrite 0x0203fd thru 0x02bc18 (0xb81c bytes) 000837 1789403286.870480 mapwrite 0x0171b8 thru 0x01815f (0xfa8 bytes) 000838 1789403286.919081 mapwrite 0x0136ce thru 0x02088c (0xd1bf bytes) 000839 1789403286.979436 write 0x034ddb thru 0x03b9e0 (0x6c06 bytes) 000841 1789403286.982593 read 0x001991 thru 0x011807 (0xfe77 bytes) 000843 1789403286.992399 mapread 0x00d3ed thru 0x00e70d (0x1321 bytes) 000846 1789403286.995241 trunc from 0x03b9e1 to 0x006883 (0x3515d bytes) 000847 1789403287.017940 mapread 0x004168 thru 0x006883 (0x271c bytes) 000849 1789403287.020461 punch from 0x0061fe to 0x006883 (0x686 bytes) 000850 1789403287.025007 trunc from 0x006884 to 0x032d8a (0x2c507 bytes) 000851 1789403287.043074 mapread 0x006dac thru 0x01050f (0x9764 bytes) 000852 1789403287.059817 write 0x01e2eb thru 0x0203a1 (0x20b7 bytes) 000854 1789403287.085164 trunc from 0x032d8b to 0x02fad9 (0x32b1 bytes) 000855 1789403287.125037 mapwrite 0x00cd1d thru 0x00f696 (0x297a bytes) 000856 1789403287.218626 punch from 0x01892b to 0x0252a7 (0xc97d bytes) 000857 1789403287.228224 MIRROR_RESYNC 6 mirrors 000858 1789403289.439838 write 0x037406 thru 0x03ffff (0x8bfa bytes) 000861 1789403289.506743 read 0x00c38f thru 0x00f526 (0x3198 bytes) 000863 1789403289.519807 mapread 0x0398ff thru 0x03f88a (0x5f8c bytes) 000864 1789403289.520779 mapread 0x0352ce thru 0x037a47 (0x277a bytes) 000865 1789403289.530486 read 0x00223d thru 0x00fbbc (0xd980 bytes) 000867 1789403289.540207 write 0x00fa10 thru 0x01a4e7 (0xaad8 bytes) 000868 1789403289.541485 mapwrite 0x037912 thru 0x03f95c (0x804b bytes) 000870 1789403289.598738 trunc from 0x040000 to 0x037468 (0x8b97 bytes) 000872 1789403289.618582 mapwrite 0x00da42 thru 0x011286 (0x3845 bytes) 000873 1789403289.666128 mapread 0x00ddc4 thru 0x0170bd (0x92fa bytes) 000874 1789403289.667953 trunc from 0x037469 to 0x01ea95 (0x189d3 bytes) 000876 1789403289.687889 write 0x023024 thru 0x031696 (0xe673 bytes) 000877 1789403289.689413 write 0x0226fd thru 0x02d464 (0xad68 bytes) 000878 1789403289.695132 trunc from 0x031697 to 0x022ac5 (0xebd1 bytes) 000879 1789403289.725347 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22ac6): Mirror 4: 0x74d7a501 (stale) Mirror 5: 0x74d7a501 (stale) Mirror 7: 0x74d7a501 (stale) Mirror 8: 0x74d7a501 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22ac6): Mirror 4: 0x74d7a501 (stale) Mirror 5: 0x74d7a501 (stale) Mirror 7: 0x74d7a501 (stale) Mirror 8: 0x74d7a501 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22ac6: skipped 000883 1789403290.114560 read 0x011c9d thru 0x018a6e (0x6dd2 bytes) 000884 1789403290.126926 mapread 0x01fa33 thru 0x020523 (0xaf1 bytes) 000886 1789403290.154430 read 0x0086e1 thru 0x011f84 (0x98a4 bytes) 000888 1789403290.175898 punch from 0x00ade1 to 0x00d300 (0x2520 bytes) 000889 1789403290.180611 mapwrite 0x01b817 thru 0x02b291 (0xfa7b bytes) 000891 1789403290.336858 write 0x037b37 thru 0x03a57e (0x2a48 bytes) 000892 1789403290.341124 read 0x02c170 thru 0x039314 (0xd1a5 bytes) 000893 1789403290.349408 trunc from 0x03a57f to 0x004a3f (0x35b3f bytes) 000894 1789403290.375759 read 0x003753 thru 0x004a3f (0x12ed bytes) 000895 1789403290.376371 mapwrite 0x03a15e thru 0x03b145 (0xfe8 bytes) 000897 1789403290.549115 write 0x03f97d thru 0x03ffff (0x683 bytes) 000898 1789403290.554225 write 0x036b6f thru 0x03bcaa (0x513c bytes) 000900 1789403290.557422 trunc from 0x040000 to 0x035234 (0xadcb bytes) 000903 1789403290.590780 punch from 0x0128b0 to 0x019e94 (0x75e5 bytes) 000904 1789403290.596183 mapwrite 0x02e625 thru 0x0354db (0x6eb7 bytes) 000906 1789403290.693148 read 0x00393e thru 0x006844 (0x2f07 bytes) 000907 1789403290.698591 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x354dc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x354dc): Mirror 4: 0xb4633b7f (stale) Mirror 5: 0xb4633b7f (stale) Mirror 7: 0xb4633b7f (stale) Mirror 8: 0xb4633b7f (stale) CRC-32 checksum value for chunk [0, 0x354dc): Mirror 1: 0xefc93bf4 Mirror 9: 0xefc93bf4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x354dc: skipped 000911 1789403291.815410 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x354dc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x354dc): Mirror 4: 0xb4633b7f (stale) Mirror 5: 0xb4633b7f (stale) Mirror 7: 0xb4633b7f (stale) Mirror 8: 0xb4633b7f (stale) CRC-32 checksum value for chunk [0, 0x354dc): Mirror 1: 0xefc93bf4 Mirror 9: 0xefc93bf4 Mirror 10: 0xefc93bf4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x354dc: skipped 000912 1789403292.985006 mapread 0x016ea8 thru 0x0267a2 (0xf8fb bytes) 000913 1789403293.021308 punch from 0x02117c to 0x027c89 (0x6b0e bytes) 000914 1789403293.115411 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x354dc): Mirror 5: 0xb4633b7f (stale) Mirror 7: 0xb4633b7f (stale) Mirror 8: 0xb4633b7f (stale) Mirror 9: 0xefc93bf4 (stale) Mirror 10: 0xefc93bf4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x354dc): Mirror 5: 0xb4633b7f (stale) Mirror 7: 0xb4633b7f (stale) Mirror 8: 0xb4633b7f (stale) Mirror 9: 0xefc93bf4 (stale) Mirror 10: 0xefc93bf4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x354dc: skipped 000915 1789403293.604980 mapread 0x00c446 thru 0x018ef7 (0xcab2 bytes) 000916 1789403293.631860 mapread 0x023333 thru 0x026118 (0x2de6 bytes) 000918 1789403293.639626 write 0x02be9c thru 0x0369e1 (0xab46 bytes) 000919 1789403293.640976 mapread 0x0009ea thru 0x00cac2 (0xc0d9 bytes) 000921 1789403293.649124 write 0x038b3d thru 0x03ffff (0x74c3 bytes) 000922 1789403293.650944 trunc from 0x040000 to 0x03098f (0xf670 bytes) 000923 1789403293.664449 read 0x01fa31 thru 0x0226c8 (0x2c98 bytes) 000924 1789403293.665320 mapwrite 0x030efd thru 0x03820b (0x730f bytes) 000925 1789403293.768555 MIRROR_RESYNC 6 mirrors 000926 1789403295.657346 write 0x00daeb thru 0x00ecf4 (0x120a bytes) 000927 1789403295.772371 mapwrite 0x024176 thru 0x02c63a (0x84c5 bytes) 000928 1789403295.936306 mapread 0x01b6f4 thru 0x01f1b4 (0x3ac1 bytes) 000929 1789403295.956150 write 0x02b74f thru 0x02fd55 (0x4607 bytes) 000930 1789403295.971703 punch from 0x00b02a to 0x01a1a8 (0xf17f bytes) 000933 1789403295.995443 punch from 0x00813a to 0x00931e (0x11e5 bytes) 000936 1789403296.003378 write 0x01f065 thru 0x02c6c9 (0xd665 bytes) 000937 1789403296.009710 mapread 0x009115 thru 0x00f037 (0x5f23 bytes) 000940 1789403296.024383 mapwrite 0x02e42e thru 0x03918e (0xad61 bytes) 000942 1789403296.123253 mapread 0x02526e thru 0x028d36 (0x3ac9 bytes) 000944 1789403296.130047 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3918f): Mirror 7: 0x34bba49d (stale) Mirror 8: 0x34bba49d (stale) Mirror 9: 0x34bba49d (stale) Mirror 10: 0x34bba49d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3918f): Mirror 7: 0x34bba49d (stale) Mirror 8: 0x34bba49d (stale) Mirror 9: 0x34bba49d (stale) Mirror 10: 0x34bba49d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3918f: skipped 000947 1789403296.554229 mapread 0x03661a thru 0x03918e (0x2b75 bytes) 000948 1789403296.587285 punch from 0x037da7 to 0x03918e (0x13e8 bytes) 000949 1789403296.595792 mapwrite 0x01b537 thru 0x02ae11 (0xf8db bytes) 000950 1789403296.724169 trunc from 0x03918f to 0x013d5e (0x25430 bytes) 000951 1789403296.749400 mapread 0x003e9c thru 0x00d65d (0x97c2 bytes) 000953 1789403296.759676 punch from 0x001f12 to 0x01074b (0xe83a bytes) 000954 1789403296.769210 MIRROR_RESYNC 5 mirrors 000955 1789403297.681293 mapread 0x003dba thru 0x00c146 (0x838d bytes) 000956 1789403297.699312 write 0x03ebbc thru 0x03ffff (0x1444 bytes) 000959 1789403297.777199 read 0x01f16f thru 0x02c458 (0xd2ea bytes) 000960 1789403297.795244 trunc from 0x040000 to 0x02e2c6 (0x11d39 bytes) 000961 1789403297.822317 punch from 0x01b1b8 to 0x01f5a4 (0x43ed bytes) 000962 1789403297.836637 punch from 0x02c24f to 0x02e2c6 (0x2078 bytes) 000963 1789403297.850445 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e2c7): Mirror 7: 0x7dafab59 (stale) Mirror 8: 0x7dafab59 (stale) Mirror 10: 0x7dafab59 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e2c7): Mirror 7: 0x7dafab59 (stale) Mirror 8: 0x7dafab59 (stale) Mirror 10: 0x7dafab59 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e2c7: skipped 000964 1789403298.184408 punch from 0x018ac4 to 0x028700 (0xfc3d bytes) 000967 1789403298.191401 mapwrite 0x019f73 thru 0x0223a5 (0x8433 bytes) 000970 1789403298.381189 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d5:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e2c7): Mirror 7: 0x7dafab59 (stale) Mirror 8: 0x7dafab59 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e2c7): Mirror 7: 0x7dafab59 (stale) Mirror 8: 0x7dafab59 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e2c7: skipped 000971 1789403298.949964 mapread 0x0267d6 thru 0x02773a (0xf65 bytes) 000972 1789403298.982230 write 0x00db93 thru 0x01cfd6 (0xf444 bytes) 000974 1789403299.000822 mapread 0x02a20d thru 0x02e2c6 (0x40ba bytes) 000976 1789403299.012504 punch from 0x01fc49 to 0x024804 (0x4bbc bytes) 000977 1789403299.060672 read 0x00db02 thru 0x014f6e (0x746d bytes) 000978 1789403299.061270 trunc from 0x02e2c7 to 0x003980 (0x2a946 bytes) 000980 1789403299.077348 write 0x007238 thru 0x012cb7 (0xba80 bytes) 000982 1789403299.080857 write 0x00f2fc thru 0x019980 (0xa685 bytes) 000983 1789403299.082779 punch from 0x00f911 to 0x0177f9 (0x7ee9 bytes) 000984 1789403299.098044 mapwrite 0x037135 thru 0x03ffff (0x8ecb bytes) 000986 1789403299.242172 write 0x001923 thru 0x00c0f7 (0xa7d5 bytes) 000987 1789403299.250307 punch from 0x00be99 to 0x011913 (0x5a7b bytes) 000988 1789403299.271539 mapread 0x03419c thru 0x038118 (0x3f7d bytes) 000989 1789403299.272612 write 0x0040f7 thru 0x0072ab (0x31b5 bytes) 000990 1789403299.278047 MIRROR_RESYNC 3 mirrors 000991 1789403300.417348 mapread 0x012c75 thru 0x01a283 (0x760f bytes) 000992 1789403300.434082 mapwrite 0x002f22 thru 0x00f656 (0xc735 bytes) 000994 1789403300.611954 mapread 0x0360c2 thru 0x03c6bf (0x65fe bytes) 000995 1789403300.622288 trunc from 0x040000 to 0x0170a1 (0x28f5e bytes) 000996 1789403300.648546 write 0x00ffae thru 0x014696 (0x46e9 bytes) 000997 1789403300.656332 read 0x006a49 thru 0x012941 (0xbef9 bytes) 000998 1789403300.656999 mapwrite 0x026203 thru 0x028110 (0x1f0e bytes) All operations completed A-OK! PASS 70a (85s) == sanity-flr test 71: check mirror extend parent fid ==== 12:28:26 (1789403306) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 12:28:30 (1789403310) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.183319 s, 11.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159555 s, 6.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x313:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x314:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x318:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x317:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x315:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 12:28:39 (1789403319) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6789440 .. Extending file size to 4183488 .. failed Extending file size to 7452800 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7701888 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 5733760 .. Extending file size to 7221792 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7957984 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9073568 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8498752 .. Extending file size to 9048928 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 57522 57523 57526 57527 57529 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) 192.168.201.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:30:03 (1789403403) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5827104 .. Extending file size to 3928576 .. done Extending file size to 7143648 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8392416 .. Extending file size to 8891840 .. Extending file size to 8844704 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9020000 .. Waiting 59180 59181 59182 59184 59187 done 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) 192.168.201.130@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:31:29 (1789403489) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107882 s, 19.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.375341 s, 5.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xfa5e49e2 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:31:49 (1789403509) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118984 s, 17.6 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 12:31:56 (1789403516) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x222:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x32a:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:32:02 (1789403522) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.32683 s, 6.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x330:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:32:12 (1789403532) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246942 s, 8.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x331:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x332:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x331:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x332:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32e:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:32:20 (1789403540) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.478363 s, 4.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x333:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x333:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x32f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x330:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:32:54 (1789403574) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg130-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134824 s, 7.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x334:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x334:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x332:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x331:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x335:0x0] } PASS 204d (58s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:33:52 (1789403632) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg130-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.360191 s, 5.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x336:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x333:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x334:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x336:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:34:52 (1789403692) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg130-server mds-ost sync done. Creating new pool test_204f oleg130-server: Pool lustre.test_204f created Adding targets to pool oleg130-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg130-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.420248 s, 5.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x337:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x335:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x337:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg130-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg130-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg130-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:36:16 (1789403776) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:36:22 (1789403782) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:36:29 (1789403789) oleg130-server: Pool lustre.test_206 created Adding targets to pool oleg130-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg130-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg130-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg130-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg130-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:36:54 (1789403814) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786389 s, 13.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414752 s, 25.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1522f4ae Mirror 3: 0x1522f4ae PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:37:04 (1789403824) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:37:08 (1789403828) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:37:12 (1789403832) Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0893871 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xd90000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x11f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x11f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x11f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1260000 Read this (4 bytes): BBBB 192.168.201.130@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg130-client.virtnet /mnt/lustre2 (opts:) PASS 209a (37s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:37:49 (1789403869) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0249714 s, 328 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:37:56 (1789403876) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548975 s, 19.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x247 lmm_fid: [0x200000404:0x247:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x247 lmm_fid: [0x200000404:0x247:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x341:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834503 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 12:37:58.000000000 -0400 Modify: 2026-09-14 12:37:58.000000000 -0400 Change: 2026-09-14 12:38:00.000000000 -0400 Birth: 2026-09-14 12:37:58.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:38:05 (1789403885) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:38:09 (1789403889) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.545552 s, 75.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0842601 s, 48.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 14 12:38 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x249:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 14 12:38 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:38:17 (1789403897) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:38:26 (1789403906) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1735 sec =========== 12:38:36 (1789403916) === sanity-flr: start cleanup 12:38:38 (1789403918) === === sanity-flr: finish cleanup 12:38:45 (1789403925) ===